Macaron: Transformer Variant & Broad Applications

• Macaron is a polysemous term defining a Transformer-derived architecture with a FFN-attention-FFN sandwich design, motivated by advanced numerical splitting methods.
• It extends into variants like Conformer blocks and CNN-based models, integrating convolution, sparse attention, and deep scaling for improved ASR and audio tasks.
• The term further applies to multilingual reasoning benchmarks and distinct systems, illustrating its evolving use across neural architecture research and applications.

Searching arXiv for recent and foundational papers on "Macaron" and "Macaron Net" to ground the article. “Macaron” is a polysemous research term whose primary technical meaning comes from Macaron Net: a Transformer-derived layer topology in which two position-wise feed-forward sublayers, each applied as a half step, sandwich self-attention (Lu et al., 2019). In later work, the same architectural motif was incorporated into Conformer-based speech recognizers, sound event detection systems, and audio masked autoencoders, while the name “Macaron” was also reused for a multilingual reasoning benchmark, a generative-UI model, and a software supply-chain security framework (Wu, 2022). The resulting literature uses “Macaron” both as a precise architectural descriptor and as a broader project name.

1. Major referents of the term

In the literature represented here, “Macaron” does not denote a single artifact. It refers to a foundational neural architecture, several derived architectural patterns, and multiple unrelated systems that share only the name.

Referent	Domain	Defining feature
Macaron Net	Sequence modeling	FFN–attention–FFN layer from an ODE-inspired derivation
Macaron-like Conformer block	ASR	Two half-step FFNs around MHSA and convolution
Convolutional Macaron Net	Sound event detection	CNN front end with a Macaron Net encoder
AudioMAE++ transformer++ block	Self-supervised audio	Macaron-style block with SwiGLU FFNs
Macaron benchmark	Multilingual evaluation	Template-first benchmark for multicultural reasoning
Macaron-A2UI / Macaron framework	UI generation / supply-chain security	Distinct systems that reuse the name

The architectural sense is historically central. Lu et al. introduced Macaron Net by arguing that the standard Transformer can be interpreted as a numerical solver for a convection-diffusion equation in a multi-particle dynamic system, and then replacing the corresponding splitting scheme with a different one that yields an FFN-attention-FFN layer (Lu et al., 2019). Later works reused “macaron-like” or “macaron-style” to describe related block structures in audio and speech models (Wu, 2022).

2. Macaron Net as an ODE-informed Transformer variant

Macaron Net was introduced in “Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View” (Lu et al., 2019). The paper maps self-attention to a diffusion term and the feed-forward network to a convection term in a multi-particle dynamic system, then interprets the standard Transformer as a Lie-Trotter splitting scheme followed by Euler’s method. The central proposal is to replace that construction with the Strang-Marchuk splitting scheme, which leads to two FFN sublayers surrounding self-attention.

The canonical Macaron layer is written as

$$\begin{align*} \tilde x_{l, i} &= x_{l, i} + \frac{1}{2} \mathrm{FFN}(x_{l, i}) \ \hat x_{l, i} &= \tilde x_{l, i} + \mathrm{MultiHeadAtt}(\tilde x_{l, i}, [\tilde x_{l, 1}, \ldots, \tilde x_{l, n}]) \ x_{l+1, i} &= \hat x_{l, i} + \frac{1}{2} \mathrm{FFN}(\hat x_{l, i}) \end{align*}$$

This “macaron-like” arrangement is explicitly motivated by the numerical-analysis claim that the standard construction corresponds to a scheme with second-order local truncation error, whereas the Strang-Marchuk replacement yields third-order local truncation error (Lu et al., 2019).

The paper also reports matched-parameter comparisons by halving the inner dimension of each FFN so that the two-FFN Macaron layer remains comparable to a standard Transformer layer. Empirically, the reported results favor Macaron Net on both supervised and unsupervised tasks: on IWSLT14 De-En, WMT14 En-De, and GLUE, the Macaron models exceed the corresponding Transformer or BERT-style baselines, with reported values of 35.4 versus 34.4 on IWSLT14 De-En, 28.9 versus 27.3 on WMT14 En-De base, 30.2 versus 28.4 on WMT14 En-De big, and 79.7 versus 77.4 on GLUE (Lu et al., 2019). In the original formulation, therefore, “Macaron” is not merely an extra FFN; it is a numerically motivated asymmetrization of the Transformer layer.

3. Conformer, deep sparse ASR, and the speech-specific macaron pattern

In speech recognition, the best-known descendant of the idea is the Conformer block, which places two macaron-like feed-forward layers around multi-head self-attention and a convolution module. “Deep Sparse Conformer for Speech Recognition” states that Conformer uses “two macaron-like feed-forward layers with half-step residual connections sandwich the multi-head self-attention and convolution modules followed by a post layer normalization” (Wu, 2022). Its block equations are

$$\begin{align} \tilde{x}_i & = x_i + 0.5\times \mathrm{FFN}(x_i) \ x'_i & = \tilde{x}_i + \mathrm{MHSA}(\tilde{x}_i) \ x''_i & = x'_i + \mathrm{Conv}(x'_i) \ y_i & = \mathrm{LayerNorm}(x''_i + 0.5\times \mathrm{FFN}(x''_i)) \end{align}$$

so the speech variant is no longer the original FFN-attention-FFN sequence, but rather FFN-attention-convolution-FFN with four residual connections (Wu, 2022).

The same paper extends this block in two directions, described as “sparser” and “deeper.” Sparse self-attention reduces time complexity and memory usage to $\mathcal{O}(L\log L)$, and a deep normalization strategy based on DeepNorm introduces depth-dependent residual scaling factors $\alpha_j$ to stabilize training of hundred-level Conformer stacks (Wu, 2022). On the Japanese CSJ-500h dataset, the reported deep sparse Conformer achieves CERs of 5.52%, 4.03%, and 4.50% on three evaluation sets, while an ensemble of five variants with 12, 16, 17, 50, and 100 encoder layers reaches 4.16%, 2.84%, and 3.20% (Wu, 2022). In this line of work, the macaron arrangement is retained structurally even as attention sparsity and depth scaling are modified.

A contrary result appears in “Squeezeformer: An Efficient Transformer for Automatic Speech Recognition” (Kim et al., 2022). That paper argues that Conformer’s design choices are “not optimal” and explicitly replaces the Macaron structure with a simpler block in which either multi-head attention or convolution is followed by a single feed-forward module. The abstract reports 7.5%, 6.5%, and 6.0% WER on LibriSpeech test-other without external LLMs, with gains of 3.1%, 1.4%, and 0.6% over Conformer-CTC at the same number of FLOPs (Kim et al., 2022). Within speech modeling, therefore, Macaron is both a successful design pattern and an object of architectural revision.

4. Convolutional Macaron Net and AudioMAE++ in audio modeling

The term was also specialized for sound event detection in “Detecting Sound Events Using Convolutional Macaron Net With Pseudo Strong Labels” (Chan et al., 2020). There, the Convolutional Macaron Net combines CNN feature extraction with a Macaron Net encoder. The encoder inserts an additional position-wise feed-forward module before multi-head attention, with each FFN scaled by 0.5:

$$\begin{align} x'_{FM} &= LN(x_{FM} + 0.5 \cdot PFF(x_{FM})) \ h_{FM} &= LN(x'_{FM} + MHA(x'_{FM})) \ y_{FM} &= LN(h_{FM} + 0.5 \cdot PFF(h_{FM})) \end{align}$$

with

$PFF(x_{FM}) = W_2(f(W_1 x_{FM} + b_1)) + b_2$

and Mish activation in the feed-forward path (Chan et al., 2020). This architecture is embedded in a semi-supervised pipeline that uses pseudo strong labels approximated using Convolutive Nonnegative Matrix Factorization and synchronously trains a frame-level model and a clip-level model with curriculum consistency costs. The reported event-based F1-score is 46.3 on validation and 46.1 on an unseen YouTube evaluation set, exceeding the DCASE 2020 baseline and slightly surpassing the top DCASE 2020 submission while using fewer Transformer encoding layers (Chan et al., 2020).

A later self-supervised audio formulation appears in “AudioMAE++: learning better masked audio representations with SwiGLU FFNs” (Yadav et al., 14 Jul 2025). AudioMAE++ replaces vanilla Transformer blocks with “transformer++ blocks” that are macaron-style in structure and combine two feed-forward sublayers with an intermediate multi-head attention layer. The first half-step uses an MLP, the second uses a SwiGLU FFN, and the architecture can optionally support rotary positional embeddings (Yadav et al., 14 Jul 2025). The pretraining pipeline divides the spectrogram into fixed non-overlapping patches, masks out 80% randomly, encodes the visible patches with transformer++ blocks, decodes with smaller transformer++ blocks, and optimizes the standard MAE mean squared error reconstruction objective (Yadav et al., 14 Jul 2025). Reported downstream results include an overall normalized score $s(m)$ of 91.8 for AudioMAE++-Base and 93.7 for AudioMAE++-Large on 10 downstream audio tasks; the paper further states that the Base model outperforms directly comparable standard MAE baselines with up to 4x more parameters (Yadav et al., 14 Jul 2025). Here, “macaron-style” remains recognizable, but the two FFN positions are no longer symmetric in implementation.

5. Macaron as a multilingual and multicultural reasoning benchmark

“Macaron: Controlled, Human-Written Benchmark for Multilingual and Multicultural Reasoning via Template-Filling” uses the name for a benchmark rather than an architecture (Elsetohy et al., 11 Feb 2026). The benchmark is template-first: 100 language-agnostic templates are defined, spanning 7 reasoning types and 22 cultural aspects, and native annotators instantiate them into scenario-aligned English and local-language multiple-choice questions plus systematically derived True/False items (Elsetohy et al., 11 Feb 2026).

The benchmark contains 11,862 instances spanning 20 countries or cultural contexts, 10 scripts, and 20 languages, including Amharic, Yoruba, Zulu, Kyrgyz, and some Arabic dialects (Elsetohy et al., 11 Feb 2026). Each base scenario produces six aligned evaluation items: English MCQ, local-language MCQ, English True, local True, English False, and local False. Evaluation is zero-shot over 21 multilingual LLMs under “Standard” and “Thinking” prompting regimes, with metrics including MCQ accuracy, T/F accuracy, paired T/F accuracy, and cross-lingual gaps (Elsetohy et al., 11 Feb 2026).

The reported aggregate results distinguish sharply between model families. Closed “Thinking” models achieve 80.8% on MC-EN, 79.7% on MC-Local, 79.3% on TF-EN, 77.4% on TF-Local, and 79.3% overall; open-weight models achieve 57.5%, 49.9%, 57.6%, 55.6%, and 55.2%, respectively (Elsetohy et al., 11 Feb 2026). The paper identifies culture-grounded mathematical and counting templates as consistently the hardest, while causal and commonsense reasoning are generally easier. Because the benchmark factorizes reasoning type and cultural aspect, it is designed as a controlled probe of multilingual reasoning rather than a translated English-centric test set (Elsetohy et al., 11 Feb 2026).

6. Generative UI and software supply-chain uses of the name

The name also appears in two technically unrelated systems. “Macaron-A2UI: A Model for Generative UI in Personal Agents” presents a model that generates natural language together with protocol-defined JSON UI actions for information collection, preference refinement, confirmation, and multi-goal organization (Kong et al., 24 May 2026). The model is trained on a large-scale Generative UI corpus built from heterogeneous dialogue datasets, uses LoRA-based supervised fine-tuning followed by Group-Relative Policy Optimization, and is evaluated on A2UI-Bench using language-side and visual-side metrics (Kong et al., 24 May 2026). The best reported model reaches 75.6 overall on A2UI-Bench without explicit schema hints, surpassing the strongest full-schema frontier baseline (Kong et al., 24 May 2026). In this usage, “Macaron” is purely the system name.

A different reuse occurs in “Unlocking Reproducibility: Automating re-Build Process for Open-Source Software,” which extends Macaron as an “industry-grade open-source supply chain security framework” to automate rebuilding Maven artifacts from source (Hassanshahi et al., 10 Sep 2025). The work is motivated by the observation that approximately 84% of the top 1200 commonly used artifacts are not built using a transparent CI/CD pipeline. The automated pipeline performs source commit resolution, build specification extraction from GitHub Actions workflows, buildspec generation, and validation (Hassanshahi et al., 10 Sep 2025). On the AROMA dataset of 473,351 artifacts, the source-detection component matches 273,811 artifacts, compared with 234,674 for AROMA; the evaluation also reports 73 successful builds on 81 reproducible packages, including 19 Gradle projects overlooked by AROMA (Hassanshahi et al., 10 Sep 2025). Here again, the shared label does not imply any relation to Macaron Net.

7. Scope, variation, and common misconceptions

A common misconception is that “Macaron” names one fixed block. The literature shows several non-identical realizations. In Macaron Net, the core layer is FFN-attention-FFN (Lu et al., 2019). In Conformer, the pattern becomes FFN-attention-convolution-FFN (Wu, 2022). In AudioMAE++, the attention layer is still sandwiched by two feed-forward paths, but the first is an MLP and the second is a SwiGLU FFN, so the two flanking sublayers are functionally differentiated (Yadav et al., 14 Jul 2025).

Another misconception is that Macaron is universally preferred wherever it appears. The evidence is mixed. Deep Sparse Conformer retains the macaron-like structure while improving long-sequence efficiency and training stability through sparse attention and DeepNorm (Wu, 2022). Squeezeformer, by contrast, explicitly drops the Macaron structure and reports better ASR accuracy-efficiency tradeoffs under the same training schemes (Kim et al., 2022). This suggests that the architectural value of Macaron is task- and system-dependent rather than unconditional.

A further point of confusion concerns the name itself. The benchmark Macaron, Macaron-A2UI, and the Macaron supply-chain framework are not presented as descendants of Macaron Net, and no shared mechanism is claimed across them (Elsetohy et al., 11 Feb 2026). A plausible implication is that “Macaron” has evolved from a precise label for an FFN-attention-FFN Transformer variant into a reusable project name across multiple subfields. Within neural architecture research, however, its most stable meaning remains the half-step feed-forward “sandwich” around a central interaction module.