Masked Audio Parallel Inference (MAPI)

• MAPI is a family of non-autoregressive audio generation methods that predict many masked tokens simultaneously and refine them iteratively.
• It leverages bidirectional models and cosine masking schedules to progressively reduce uncertainty, enabling efficient semantic-to-acoustic mapping.
• Systems like SoundStorm, SpecMaskGIT, and IMPACT use MAPI, achieving significantly faster and high-quality audio synthesis compared to autoregressive approaches.

Searching arXiv for primary and related papers on masked parallel audio inference. Masked Audio Parallel Inference (MAPI) denotes a family of non-autoregressive audio generation procedures in which many masked audio tokens or latent variables are predicted simultaneously and refined over a small number of iterations, rather than generated strictly left-to-right or through hundreds of diffusion steps. In the narrowest and most concrete sense, MAPI is the specific masked, confidence-based parallel decoding procedure used by SoundStorm to map semantic tokens to SoundStream codec tokens (Borsos et al., 2023). In a broader usage that emerges across later work, the term refers to a design pattern: initialize audio representations in a maximally or partially masked state, use a bidirectional or masked model to predict missing content in parallel, and progressively reduce uncertainty through schedule-controlled refinement in discrete or continuous latent spaces (Comunità et al., 2024).

1. Definition and scope

In SoundStorm, audio generation is organized as a two-stage semantic-to-acoustic pipeline. The input is a time-aligned sequence of semantic tokens, and the output is a tensor of SoundStream codec tokens. MAPI is the inference procedure that begins from a fully masked codec-token sequence, except for optional prompt frames, decodes all frames of one residual vector-quantization (RVQ) level in parallel, and then moves to the next level. Within each RVQ level, it performs iterative masked prediction: the model predicts all masked positions, fixes the most confident tokens, and keeps uncertain positions masked for further refinement. This is the mechanism by which SoundStorm replaces the autoregressive acoustic stages of AudioLM while remaining non-autoregressive and parallel (Borsos et al., 2023).

Subsequent papers use closely related formulations. SpecMaskGIT describes MAPI as the idea of generating or transforming audio by predicting many masked latent audio tokens in parallel over a few iterative refinement steps, rather than stepping through a waveform or latent sequence autoregressively or with hundreds of diffusion steps. IMPACT implements an iterative, mask-based, non-autoregressive decoding scheme in a continuous latent space with diffusion modeling. AudioMNTP does not name its method MAPI, but its masked next-token prediction framework is explicitly designed to enable parallel masked audio inference. This suggests that MAPI is best understood not as a single standardized algorithm, but as a family of masked, iterative, parallel decoding schemes adapted to different audio representations and conditioning regimes (Comunità et al., 2024).

A common misconception is to equate MAPI with any masked-audio model. The literature distinguishes between masked reconstruction for representation learning, masked parallel generation, and masked post-hoc interpretation. MaskSpec reconstructs masked spectrogram patches in parallel but is presented as self-supervised pre-training rather than synthesis, whereas SoundStorm, SpecMaskGIT, IMPACT, and G-MLM/G-IPD use masked prediction directly as a generation procedure (Chong et al., 2022).

2. Canonical formulation in SoundStorm

SoundStorm provides the most explicit and operational definition of MAPI. Its conditioning sequence consists of semantic tokens $S$, derived by quantizing w2v-BERT or BEST-RQ embeddings with k-means. In the AudioLM experiments, the semantic stream runs at 25 tokens per second with a codebook size of 1024 clusters, and the tokens are duplicated to reach 50 tokens per second so that they match the codec frame rate. The target acoustic representation is a SoundStream RVQ tensor

$Y \in \{1,\dots,C\}^{T \times Q},$

with $Q = 12$ RVQ levels, $C = 1024$ per-level codewords, and 50 frames per second, for an overall bitrate of

$50 \cdot 12 \cdot \log_2(1024) = 6000\ \text{bps}.$

Semantic tokens encode high-level linguistic or semantic content and coarse prosody, while codec tokens encode full acoustic detail, including voice timbre, background, and fine prosody (Borsos et al., 2023).

The model architecture is a bidirectional Conformer with 12 layers, 350M parameters, 16 attention heads, embedding size 1024, FF dimension 4096, convolution kernel size 5, and rotary positional embeddings. For each frame, SoundStorm sums embeddings for the semantic token and all codec-level tokens at that frame, using a dedicated mask embedding for masked codec positions, so the attention sequence length depends on $T$ rather than $T \times Q$. After the Conformer, $Q$ separate dense heads produce per-level distributions over the codebook. This RVQ-aware construction is central: it reduces the effective attention length and lets the model predict per-level tokens while using non-causal bidirectional context (Borsos et al., 2023).

Training masks are designed to imitate inference. A prompt delimiter $t$ is sampled uniformly, prompt frames up to $t$ are never masked, an RVQ level $Y \in \{1,\dots,C\}^{T \times Q},$0 is sampled uniformly, and a cosine masking ratio $Y \in \{1,\dots,C\}^{T \times Q},$1 with $Y \in \{1,\dots,C\}^{T \times Q},$2 determines which positions in level $Y \in \{1,\dots,C\}^{T \times Q},$3 are masked. All finer levels $Y \in \{1,\dots,C\}^{T \times Q},$4 are entirely masked for non-prompt frames, and the cross-entropy loss is computed only on masked tokens at level $Y \in \{1,\dots,C\}^{T \times Q},$5. At inference, SoundStorm decodes levels in coarse-to-fine order. Level 1 uses 16 iterations of masked decoding, while levels 2 through 12 use 1 iteration each, for 27 forward passes total. Within a level, the model predicts all masked positions, ranks them by confidence, fixes a fraction according to a cosine schedule, and in the last iteration uses greedy decoding rather than sampling because this improves audio quality (Borsos et al., 2023).

3. Representation choices and architectural variants

Later MAPI-style systems differ chiefly in the latent representation on which masking and parallel refinement operate.

System	Audio representation	Masked parallel inference form
SoundStorm	SoundStream codec tokens with $Y \in \{1,\dots,C\}^{T \times Q},$6 RVQ levels at 50 frames/s	Level-wise iterative masked decoding; 16 iterations for level 1 and 1 iteration for levels 2–12
SpecMaskGIT	265 SpecVQGAN tokens for a 10 s Mel-spectrogram clip	16-iteration MaskGIT-style refinement with cosine re-masking and Gumbel-noised confidence
IMPACT	Continuous VAE latent sequence $Y \in \{1,\dots,C\}^{T \times Q},$7	Cosine-scheduled monotonic unmasking with diffusion reverse steps on selected positions
G-MLM/G-IPD	HiFi-Codec bi-group, bi-depth G-RVQ tokens	Group-wise iterative coarse decoding plus single-step fine decoding

SpecMaskGIT works on a latent Mel-spectrogram space. A 10 s mono audio clip at 22.05 kHz is converted to a Mel-spectrogram with 848 time frames and 80 Mel bins, then tokenized by a modified SpecVQGAN operating on non-overlapping $Y \in \{1,\dots,C\}^{T \times Q},$8 time-Mel patches. Each patch maps to a codebook of size 1024 with 256-dimensional embeddings, yielding 265 discrete tokens per 10 s clip. The generative backbone is a bidirectional ViT-style Transformer with 24 blocks, attention dimension 768, 8 heads, and FFN dimension 3072, conditioned by a CLAP embedding token and optionally by a conditional mask token that replaces the standard learned mask token at masked positions (Comunità et al., 2024).

IMPACT moves MAPI into a continuous latent domain. It starts from the AudioLDM VAE latent tensor $Y \in \{1,\dots,C\}^{T \times Q},$9, applies a patching factor $Q = 12$0, and flattens to a sequence $Q = 12$1, where $Q = 12$2. A transformer-based latent encoder receives projected text conditioning from CLAP and Flan-T5 together with the visible latent positions, then a lightweight MLP diffusion head predicts denoising noise for masked positions. A crucial design choice is that the heavy transformer sits outside the diffusion loop, while the inexpensive denoiser sits inside it (Huang et al., 31 May 2025).

AudioMNTP also uses continuous-valued tokens derived from the AudioLDM latent space, but it keeps a decoder-only causal language-model factorization. After VAE encoding and patchification, a 10 s clip becomes a sequence of $Q = 12$3 continuous tokens of dimension $Q = 12$4. The next-token distribution is modeled with token-wise diffusion rather than categorical prediction, and masked next-token prediction drops tokens from the context sequence and augments the model with target positional embeddings so that it can learn to predict arbitrary future positions under sparse past context. The paper states that, theoretically, one can devise a parallel decoding algorithm by manipulating the target positional embedding (Yang et al., 14 Jul 2025).

A distinct discrete-token variant appears in “Efficient Parallel Audio Generation using Group Masked Language Modeling,” which combines HiFi-Codec with bi-group, bi-depth Group-RVQ, semantic tokens from Wav2Vec 2.0 XLSR clustered into 512 units, and a conformer prediction network with cross-attention to a prompt encoder. Its masking and inference procedures operate over grouped codec streams rather than the 12-level SoundStream layout used by SoundStorm (Jeong et al., 2024).

4. Decoding mechanics

Across MAPI systems, the basic inference pattern is iterative masked prediction under a shrinking mask schedule. In SoundStorm, the sequence is initialized with prompt codec frames, if any, and all non-prompt RVQ positions are set to $Q = 12$5. For a current level $Q = 12$6, the model computes distributions $Q = 12$7 for all currently masked positions, uses a confidence score such as the maximum predicted probability, fixes a fraction of positions according to a cosine schedule, and revisits the remainder. The last iteration within a level is greedy. Because attention is bidirectional rather than causal, every frame can use past and future context, and all masked positions in the level share the same forward pass (Borsos et al., 2023).

SpecMaskGIT keeps the same overall logic but changes the update rule. Training masks a random fraction $Q = 12$8 drawn from a truncated Gaussian centered at 55%, and the loss is cross-entropy with label smoothing of 0.1 on masked positions only. Inference starts from a fully masked or partially masked token sequence, appends a CLAP condition token, computes conditional and unconditional logits for classifier-free guidance,

$Q = 12$9

samples categorical codes for all masked positions in parallel, and then re-masks the lowest-confidence positions according to a cosine schedule. Confidence includes Gumbel perturbations,

$C = 1024$0

with linearly annealed $C = 1024$1, so early iterations are more stochastic and late iterations more deterministic (Comunità et al., 2024).

IMPACT preserves iterative masking but replaces categorical prediction with diffusion in a continuous latent space. During training, a random mask hides a fraction $C = 1024$2 of latent positions, the encoder processes text together with unmasked latents, and the diffusion head predicts denoising noise only for masked positions. At inference, the latent sequence starts fully masked, a cosine schedule

$C = 1024$3

determines how many positions remain masked, and a monotonic subset constraint ensures that once a position is unmasked it is never re-masked. Because continuous latents do not provide discrete confidence scores, IMPACT randomly selects which masked positions are revealed at each iteration, then performs diffusion reverse steps only for those positions, using classifier-free guidance scheduled by a cosine with $C = 1024$4 (Huang et al., 31 May 2025).

Group-Masked Language Modeling and Group Iterative Parallel Decoding retain confidence-based iterative refinement in a discrete codec setting, but the unit of refinement is a group-wise coarse token space. Coarse tokens across both groups are initialized as mask tokens, the model predicts all remaining coarse tokens in parallel, top-confidence positions across the joint group space are fixed under a cosine masking schedule, and fine tokens are then generated in a single non-iterative step conditioned on the finalized coarse tokens. The prompt voice is injected by multi-head cross-attention from a prompt encoder, and the prompt keys and values can be cached across iterations (Jeong et al., 2024).

5. Empirical characteristics and applications

The most direct evidence for MAPI’s efficiency comes from SoundStorm. With 50 frames per second, 12 RVQ levels, and 18,000 codec tokens for 30 s of audio, SoundStorm uses 27 forward passes, generates 30 seconds of audio in 0.5 seconds on a TPU-v4 including SoundStream decoding to waveform, and reaches a real-time factor of $C = 1024$5, or 58× faster than real time. Compared with AudioLM’s acoustic generator, it is two orders of magnitude faster. On LibriSpeech test-clean, SoundStorm improves WER and CER over AudioLM across short, mid, and long segments, improves WavLM-based speaker similarity in the prompted setting from $C = 1024$6 to $C = 1024$7, and keeps acoustic consistency scores close to original audio at $C = 1024$8 versus AudioLM’s $C = 1024$9. The same system is used for 30 s dialogue synthesis from transcripts annotated with speaker turns and short voice prompts, with total runtime around 2 s for semantic generation, SoundStorm decoding, and codec waveform synthesis (Borsos et al., 2023).

SpecMaskGIT demonstrates that MAPI-style synthesis is not limited to codec tokens. It synthesizes a realistic 10 s audio clip by less than 16 iterations, is real-time with only 4 CPU cores, and is about 30× faster than real time on an RTX A6000. On AudioCaps, SpecMaskGIT with BigVSAN achieves FAD 2.7 using 170M parameters and 16 iterations, compared with DiffSound at 7.8 using 400M parameters and 100 iterations, MAGNeT-small at 3.2 using 300M parameters and 180 iterations, and AudioGen-base at 3.1 using 285M parameters and 500 iterations. The same masked generative core performs zero-shot time inpainting and bandwidth extension without task-specific fine-tuning, improving FAD from 1.6 to 1.2 in the inpainting setting and from 2.7 to 1.5 for bandwidth extension, or to 0.4 with low-frequency replacement (Comunità et al., 2024).

IMPACT shows that masked parallel decoding can be combined with continuous latent diffusion. On AudioCaps, with 16 decoding iterations and 100 diffusion steps, the base model reaches latency 5.7 s with FAD 1.71, KL 1.04, IS 10.17, and CLAP 0.361 on a single Tesla V100 at batch size 8. The broader latency-quality curve shows FAD improving from 5.75 at 2 decoding iterations to 1.07 at 32 iterations, then slightly worsening to 1.13 at 64 iterations. At batch size 64 and 4 decoding iterations, throughput is about 8.26 samples/s for IMPACT versus about 1.23 samples/s for MAGNET-S (Huang et al., 31 May 2025).

The group-wise codec variant also reports strong prompt-based speech results. On LibriTTS test-clean with the same prompt speaker, the proposed model at $50 \cdot 12 \cdot \log_2(1024) = 6000\ \text{bps}.$0 attains CER 2.69, SECS 0.475, MOS 4.06, and SMOS 4.30, while SoundStorm at $50 \cdot 12 \cdot \log_2(1024) = 6000\ \text{bps}.$1 reports CER 2.87, SECS 0.437, MOS 3.83, and SMOS 4.10. With a different prompt speaker, the proposed model at $50 \cdot 12 \cdot \log_2(1024) = 6000\ \text{bps}.$2 reports CER 2.49, SECS 0.416, MOS 4.38, and SMOS 4.51, compared with SoundStorm at $50 \cdot 12 \cdot \log_2(1024) = 6000\ \text{bps}.$3 with CER 3.11, SECS 0.392, MOS 3.86, and SMOS 4.24. The paper also states that at $50 \cdot 12 \cdot \log_2(1024) = 6000\ \text{bps}.$4, its model surpasses SoundStorm with $50 \cdot 12 \cdot \log_2(1024) = 6000\ \text{bps}.$5, and that even at $50 \cdot 12 \cdot \log_2(1024) = 6000\ \text{bps}.$6, quality remains superior (Jeong et al., 2024).

AudioMNTP provides a different empirical perspective: its masked training objective improves causal generation even before a parallel decoding algorithm is implemented. On AudioCaps, AudioMNTP Base improves over AudioNTP Base from FAD 2.28 to 1.68, KL from 1.29 to 1.16, and CLAP from 0.308 to 0.336. AudioMNTP Large reaches FAD 1.22, FD 14.30, KL 1.17, IS 9.81, and CLAP 0.341 with 462M parameters, while remaining streamable because it retains a decoder-only language-model architecture (Yang et al., 14 Jul 2025).

6. Position in the literature, limitations, and open questions

MAPI is closely related to masked generative modeling more broadly, but the audio literature shows that its exact operational meaning depends on the representation. In spectrogram-space self-supervision, MaskSpec removes random spectrogram patches from the encoder input and reconstructs all masked patches in parallel with an asymmetric transformer encoder-decoder and an MSE loss. It uses $50 \cdot 12 \cdot \log_2(1024) = 6000\ \text{bps}.$7 patches, 496 patches per spectrogram, and achieves strong downstream performance after pre-training on AudioSet, including 0.471 mAP on AudioSet, 0.854 mAP on OpenMIC2018, 0.982 accuracy on ESC-50, 0.976 accuracy on SCV2, and 0.823 accuracy on DCASE2019 Task1A. This suggests that parallel masked reconstruction was an important precursor for later generative MAPI systems, even though the task was framed as self-supervised audio pre-training rather than synthesis (Chong et al., 2022).

The main limitations reported for MAPI-style generation are representation bottlenecks, iteration trade-offs, and uncertainty management. In SoundStorm, 16 iterations for level 1 yield a 0.1–0.2 MOS improvement over pure greedy level-wise decoding, but increasing beyond 16 does not significantly improve quality, and very complex non-speech sounds or music may require more refinement in finer RVQ levels. In SpecMaskGIT, the tokenizer is the bottleneck: the gap between vocoder-only and full-pipeline rFAD indicates that improving SpecVQGAN could further enhance quality, and continuous diffusion models still achieve lower FAD. In IMPACT, continuous latent confidence is not exploited; random selection under a cosine schedule works, but the paper identifies adaptive masking and fewer diffusion steps as open directions. In AudioMNTP, a practical parallel decoding algorithm is not yet implemented; the paper states only that such an algorithm can theoretically be devised by manipulating target positional embeddings (Borsos et al., 2023).

A final misconception is that MAPI is inherently tied either to discrete codec tokens or to bidirectional models. The surveyed systems collectively refute both restrictions. SoundStorm and G-MLM/G-IPD are discrete and bidirectional; SpecMaskGIT is discrete but spectrogram-based; IMPACT is continuous and bidirectional at the encoder with diffusion-based masked refinement; AudioMNTP is continuous but causal, and its masked next-token prediction objective is explicitly designed to support future parallel masked inference. A plausible implication is that MAPI should be regarded less as a single architecture than as an inference principle: parallel prediction of missing audio structure under a masking schedule, with the exact implementation determined by the audio latent space, the conditioning path, and the mechanism used to decide which masked positions are fixed at each stage (Yang et al., 14 Jul 2025).