Falcon3-Audio: Open Public ALMs

• The paper introduces Falcon3-Audio as a family of open, public-data-only audio-language models that use a minimalist, single-stage training approach to achieve competitive performance.
• Falcon3-Audio integrates a pretrained Whisper encoder, an instruction-tuned Falcon3 LLM, and a lightweight projector, bypassing complex multi-stage pipelines and elaborate cross-modal fusion.
• The model demonstrates strong performance on general sound and speech tasks while exposing limitations in music reasoning and multilingual capability, emphasizing efficiency and reproducibility.

Searching arXiv for the specified paper and closely related audio-LLM references. Falcon3-Audio is a family of open, public-data-only audio-LLMs (ALMs) introduced in "Competitive Audio-LLMs with Data-Efficient Single-Stage Training on Public Data" (Kumar et al., 9 Sep 2025). It is defined as an audio-conditioned instruction-following LLM that accepts an audio input together with a text instruction or prompt and generates a text answer. The model family is positioned as a general ALM spanning speech, music, and environmental sound rather than a system specialized only for ASR or a single audio domain. Its central claim is that strong general audio understanding does not require proprietary data, multi-stage curricula, multiple audio encoders, or complicated cross-attention fusion; instead, a pretrained Whisper encoder, an instruction-tuned Falcon3 LLM, a lightweight projector, and single-stage supervised fine-tuning with LoRA are reported to be sufficient for competitive performance (Kumar et al., 9 Sep 2025).

1. Conceptual framing and motivation

Falcon3-Audio was proposed against a background in which audio-language modeling had, according to the paper, lagged behind vision-language modeling and become fragmented by increasingly complex design recipes. The paper identifies several recurrent ingredients in prior ALM practice: multi-stage pipelines, large proprietary or undisclosed datasets, multiple audio encoders, elaborate cross-modal attention or connector modules, and RL or curriculum stages layered on top of supervised training. Falcon3-Audio is presented as a transparent baseline intended to isolate what is actually necessary for strong performance (Kumar et al., 9 Sep 2025).

In that framing, the model family serves two purposes. First, it is a practical ALM for instruction-following over heterogeneous audio inputs. Second, it is an empirical argument about methodological minimalism: the reported results are used to support the claim that a comparatively simple architecture and a single end-to-end training stage can remain highly competitive. This suggests that part of the recent complexity in ALM development may reflect convention rather than necessity, at least on the benchmark suite emphasized in the paper.

A common misconception addressed implicitly by the work is that strong open-weight ALMs must depend on either very large-scale proprietary corpora or a stack of alignment stages. The reported Falcon3-Audio results are used to challenge that assumption by pairing public-data provenance with comparatively small training exposure and a supervised-only optimization pipeline (Kumar et al., 9 Sep 2025).

2. Architecture and model family

Falcon3-Audio is released in three sizes: Falcon3-Audio 1B, Falcon3-Audio 3B, and Falcon3-Audio 7B. The parameter breakdown reported in the paper is summarized below.

Variant	Audio Encoder	Projector	LLM	Total
7B	307M	12M	7.4B	7.8B
3B	307M	12M	3.2B	3.6B
1B	88M	5M	1.7B	1.8B

The language backbone is the Falcon3 Instruct family rather than base LLMs. The paper states that instruction-tuned LLMs were chosen because ALM tasks are typically phrased as instructions and questions, and the experiments reported there indicated that this choice was better. For the 7B variant, the model uses an internal Falcon3 7B instruct version with less code and math emphasis than the publicly released one, because that variant worked better in the reported setting (Kumar et al., 9 Sep 2025).

The audio encoder is Whisper. The 7B and 3B variants use Whisper Medium English, while the 1B variant uses Whisper Small English. The paper states that Whisper was chosen because it is especially strong for speech while also remaining robust for non-speech audio, and that the authors’ ablations found it superior to larger or multilingual Whisper variants for the benchmark mix under study.

The overall architecture is deliberately simple. Audio is encoded with Whisper, the resulting audio-token sequence is downsampled or pooled, the audio features are projected into the LLM embedding space, and the projected audio embeddings are concatenated with the text-token embeddings before autoregressive generation. The paper characterizes the path as Whisper audio encoder $\rightarrow$ learnable projector $\rightarrow$ instruction-tuned Falcon3 LLM (Kumar et al., 9 Sep 2025).

The projector is defined as a two-layer mapping with LayerNorm, GELU, and a final LayerNorm: $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$

$h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$

$h_{\text{act}} = \sigma(h_{\text{proj1}})$

$h_{\text{proj2}} = W_2 h_{\text{act}} + b_2$

$$h_{\text{final}} = \text{LayerNorm}(h_{\text{proj2}})$$

where $\sigma$ is GELU. The multimodal input to the LLM is then given as

$$h_{\text{input}} = [h_{\text{final}}, \text{EMB}(z)]$$

with $z$ denoting the tokenized text prompt or instruction. Conditioned on audio $\rightarrow$0 and text prompt $\rightarrow$1, the LLM produces output text $\rightarrow$2 via

$\rightarrow$3

A central architectural point is what Falcon3-Audio does not include. The model does not use inserted cross-attention layers inside the LLM, multi-encoder fusion, intermediate-layer feature aggregation from the encoder, or complicated temporal aggregators. The paper treats this omission not as a limitation but as a design thesis: a single projector into the input embedding space is argued to be sufficient for strong ALM performance in this setup (Kumar et al., 9 Sep 2025).

3. Training pipeline, data mixture, and implementation

Falcon3-Audio is trained in a single end-to-end supervised fine-tuning stage. The training set is described as

$\rightarrow$4

where Open-ASQA contributes about 10 million samples and 26K hours, and a synthetic voice-instruction dataset contributes 440K samples and approximately 1K hours. Total audio exposure is therefore roughly 27K hours, rounded in the paper as less than 30K hours, while the abstract emphasizes that this corresponds to only about 5K unique hours because many instruction examples are alternative formulations over the same underlying audio (Kumar et al., 9 Sep 2025).

Open-ASQA is described as being built from public sources including AudioSet, AudioCaps, FSD50K, FreeSound, LibriTTS, VGGSound, and VoxCeleb. The added instruction dataset is identified as Instruction Speech or voice-instruction data. The paper explicitly states that adding more public datasets in the reported experiments did not improve target benchmark performance; a 10K-hour LibriSpeech subset, Auto-ACD, or both together failed to help meaningfully.

On the audio side, the paper relies on the Whisper encoder pipeline rather than specifying raw waveform preprocessing minutiae. What is stated explicitly is the sequence rate: Whisper downsampling yields roughly 50 tokens per second, and Falcon3-Audio applies a simple pooling layer to reduce this to 25 tokens per second. The language side uses Falcon3 tokenization. The prompt format includes the special placeholder token <|AUDIO|>, and the supplementary figure states that this token is replaced with audio features during fine-tuning.

The training objective is standard autoregressive negative log-likelihood over completion tokens: $\rightarrow$5 The projector parameters are trained directly, while both the audio encoder and the LLM are adapted with LoRA. The reported LoRA hyperparameters are rank $\rightarrow$6, alpha $\rightarrow$7, and dropout $\rightarrow$8. The paper also notes that DoRA was tried and performed worse (Kumar et al., 9 Sep 2025).

The optimization setup is fully specified in the supplied technical summary: 1 epoch, BF16, maximum sequence length 4096, per-device train batch size 4, global train batch size 256, gradient accumulation steps 1, maximum learning rate $\rightarrow$9, cosine scheduler, warmup ratio 0.01, weight decay 0.01, maximum gradient norm 1.0, and AdamW. Padding is described as audio padding to max length with global padding by longest on the right.

Implementation details are unusually explicit. The training stack uses HuggingFace Transformers, specifically the Qwen2AudioForConditionalGeneration class, together with HuggingFace Accelerate, DeepSpeed ZeRO-2, and TRL SFT Trainer. The hardware is AWS SageMaker on 8 P5 nodes, each with $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$0GB H100 GPUs, for a total of 64 H100 GPUs, and the reported training time is approximately 20 hours (Kumar et al., 9 Sep 2025). A plausible implication is that the paper is attempting not only to optimize for performance but also to reduce the reproducibility barrier that often surrounds ALM systems.

4. Benchmark performance and empirical profile

The headline result is on MMAU test, where Falcon3-Audio-7B scores 64.14. The same table reports 57.96 for Falcon3-Audio-3B and 50.80 for Falcon3-Audio-1B (Kumar et al., 9 Sep 2025).

Model	Size	Train data	MMAU Avg
Falcon3-Audio 7B	7.8B	0–30K h	64.14
Falcon3-Audio 3B	3.6B	0–30K h	57.96
Falcon3-Audio 1B	1.8B	0–30K h	50.80

The category breakdown for Falcon3-Audio-7B is 71.27 on Sound, 58.53 on Music, and 62.63 on Speech. The paper compares this to R1-AQA, which is reported at 64.36 overall, yielding a gap of 0.22 points. It further contrasts Falcon3-Audio with Qwen2-Audio Instruct at 52.50 and Audio Flamingo 2 at 59.42, emphasizing that Falcon3-Audio-7B exceeds those reported scores while using a simpler connector and much less training data. Falcon3-Audio-1B is highlighted as remaining competitive with larger open models in the 2B–13B range (Kumar et al., 9 Sep 2025).

The task-category profile is asymmetric. Falcon3-Audio-7B is strongest in Sound and Speech, and the paper reports that its Sound score of 71.27 is higher than R1-AQA’s 69.76. By contrast, Music is a relative weakness: the 58.53 Music score trails both R1-AQA at 61.40 and Audio Flamingo 2 at 72.90. This suggests that Falcon3-Audio’s reported strengths lie more in general sound and speech understanding than in top-tier music reasoning.

On AIR-Bench Foundational, Falcon3-Audio-7B scores 54.00 overall, ahead of Qwen2-Audio Instruct at 44.00. The 3B and 1B variants score 42.00 and 38.00 respectively. Category averages for the 7B model are 65.2 on Sound, 50.5 on Music, and 50.4 on Speech. Notable task-level values reported for Falcon3-Audio-7B include 70.70 on Audio grounding, 90.70 on Vocal sound classification, 71.60 on Sound QA, 66.00 on Spoken language identification, and 64.10 on Emotion recognition. The paper also notes tasks where Qwen2-Audio does better, such as Intent classification and Speech entity recognition, while maintaining that Falcon3-Audio wins overall (Kumar et al., 9 Sep 2025).

AIR-Bench Foundational is accompanied by an evaluation caveat. The authors report modifying the original scorer because, in their assessment, it could accept wrong answers if the option letter matched, ignore correct textual answers that omitted the option identifier, and ignore non-generations rather than count them wrong. Their revised scorer matches answer content textually and counts invalid outputs as incorrect. This is not merely an implementation detail; it affects interpretation of the reported numbers and constitutes one of the paper’s explicit methodological interventions.

On AIR-Bench Chat, which uses GPT-4-Turbo as evaluator with temperature 1.0, a 50% chance of switching to 2.0, and median score over three trials, Falcon3-Audio-7B scores 6.20. The paper reports 6.93 for Qwen2-Audio Instruct and 6.98 for Phi-4-Mini, placing Falcon3-Audio-7B third among the end-to-end open-weight ALMs listed there. The generation settings for Falcon3-Audio on this benchmark are the MMAU settings plus repetition penalty 1.1 and max new tokens 512 (Kumar et al., 9 Sep 2025). The authors argue that GPT-4-as-judge may under-credit semantically correct but stylistically different answers, which the paper presents as a possible evaluator bias rather than a definitive flaw in the benchmark.

5. Ablations, simplification claims, and efficiency arguments

The ablation study, run on Falcon3-Audio 7B on MMAU test-mini, is central to the paper’s broader methodological claim. The base model in that setting achieved Total 65.0, Sound 69.4, Music 62.3, and Speech 63.0. Against that baseline, two-stage curriculum training produced a total change of $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$1, with declines across all three subcategories. Within the reported setup, this supports the claim that single-stage training is not only simpler but better (Kumar et al., 9 Sep 2025).

Intermediate Whisper feature aggregation also performed worse than using final-layer features alone. Before the projector, extracting every 12, 6, or 3 layers yielded total changes of $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$2, $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$3, and $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$4. After the projector, the corresponding changes were $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$5, $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$6, and $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$7. The paper therefore concludes that final-layer Whisper features are sufficient in this design.

Stronger temporal compression was likewise unnecessary. With stack factor 2, total accuracy changed by $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$8, though with tradeoffs across Sound, Music, and Speech. Stack factor 4 produced $$h_{\text{norm1}} = \text{LayerNorm}(h_{\text{audio}})$$9, and stack factor 8 produced $h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$0. The reported pattern is that modest compression may be tolerable, but aggressive reduction degrades performance.

Encoder scaling did not behave monotonically. Relative to Whisper Medium English, Whisper Small gave $h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$1, Whisper Large $h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$2, Whisper Turbo $h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$3, and Whisper Medium Multilingual $h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$4, with the multilingual medium model especially hurting speech by $h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$5. In the paper’s interpretation, the most appropriate encoder for the benchmark composition was not the largest one but the English-medium variant.

Data scaling beyond the chosen mixture also failed to help meaningfully. Adding a LibriSpeech 10K subset resulted in $h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$6, adding Auto-ACD $h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$7, and adding both $h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$8. This is one of the sharpest empirical supports for the paper’s data-efficiency claim: more public data was not automatically better in the reported configuration.

The strongest ablation concerns joint adaptation. Freezing only the audio encoder caused $h_{\text{proj1}} = W_1 h_{\text{norm1}} + b_1$9, freezing only the LLM caused $h_{\text{act}} = \sigma(h_{\text{proj1}})$0, and freezing both caused $h_{\text{act}} = \sigma(h_{\text{proj1}})$1. The paper uses these numbers to argue that the best strategy is to train the projector while adapting both audio encoder and LLM with LoRA. Freezing the LLM is described by the results as catastrophic, implying that language-side adaptation is essential for effective audio conditioning (Kumar et al., 9 Sep 2025).

These ablations underpin the broader efficiency argument. Falcon3-Audio-7B uses less than 30K total public training hours and about 5K unique hours, yet reaches 64.14 on MMAU. The paper contrasts this with Qwen2-Audio Instruct at 52.50 trained on 500K–2M hours, R1-AQA at 64.36 built on the same Qwen2-Audio pipeline with an added GRPO reinforcement-learning stage over 38K aligned samples, Phi-4-Mini at 55.56 trained on 2M+ multimodal hours, and Audio Flamingo 2 at 59.42 trained on 30K–100K hours. The paper’s quantitative framing is that Falcon3-Audio is within 0.22 points of the top reported open-weight score while using roughly 17x to 67x less total audio data than the 500K–2M-hour class of models (Kumar et al., 9 Sep 2025).

Parameter efficiency is also emphasized. Falcon3-Audio-7B totals 7.8B parameters, smaller than the 8.4B R1-AQA / Qwen2-Audio Instruct systems and markedly smaller than SALMONN at 13B. The 1B variant totals 1.8B and attains 50.80 MMAU, which the paper characterizes as competitive with many 2B–13B open-weight models. The authors also note more than 90% overlap in training data with LTU-AS but a much higher MMAU score, 64.14 versus 18.90, using that comparison to argue that architectural and training choices, rather than mere data overlap, explain much of the gain.

6. Limitations, evaluation caveats, and practical positioning

The paper is explicit that Falcon3-Audio is constrained by a public-data-only philosophy. This is treated simultaneously as a transparency advantage and a scale limitation. Future work is said to require more diverse data, more multilingual audio, larger models, and RL-based alignment. Falcon3-Audio should therefore not be interpreted as a claim that such extensions are universally unnecessary; rather, the reported claim is that they were not required to achieve the stated benchmark competitiveness in the present setup (Kumar et al., 9 Sep 2025).

The model is also English-heavy. The chosen encoder for the larger variants is Whisper Medium English, and the paper notes that benchmark speech content is largely English. The multilingual Whisper variant performed worse in the reported ablations, but the paper indicates that this likely reflects benchmark composition rather than a general multilingual disadvantage. A plausible implication is that the present model family should not be read as a multilingual ALM solution.

Music understanding is a further limitation. Falcon3-Audio is reported as strong overall, but its MMAU Music score of 58.53 trails stronger competitors, including R1-AQA and especially Audio Flamingo 2. The same relative weakness appears on AIR-Bench Chat, where the Music and Mixed scores lag those of stronger chat-oriented systems. For workloads centered on music reasoning, the reported evidence suggests that Falcon3-Audio is competent but not dominant.

The paper also flags sensitivity in GPT-4-as-judge evaluation. It presents qualitative cases where Falcon3-Audio allegedly receives lower scores for concise, paraphrased, or stylistically different answers even when the content is semantically correct. This does not invalidate AIR-Bench Chat, but it does mark an important interpretive caveat: benchmark outcomes on open-ended judged tasks may partly reflect evaluator preference for particular response styles.

Finally, the work is more transparent than many competitors without being a complete line-by-line training cookbook. The paper does not fully specify exact raw audio preprocessing parameters, exact prompt text template contents beyond the <|AUDIO|> placeholder figure, exact LoRA target modules, or exact inference mechanics for long-audio chunking beyond pooled token rates. For researchers and practitioners, Falcon3-Audio is therefore best understood as a reproducibility-oriented ALM baseline with unusually detailed reporting, rather than as an exhaustively specified recipe (Kumar et al., 9 Sep 2025).

In practical terms, the paper implies that Falcon3-Audio is most suitable when open weights, public-data provenance, comparatively reproducible training, and strong general audio QA or instruction-following matter more than maximum music specialization or multilingual coverage. Within the family, the 7B model is the flagship accuracy point, while the 1B model is particularly notable for compact deployment settings because it remains competitive despite its substantially smaller size.