FRAME: Fractional-Fourier Mixture of Experts

• The paper introduces FRAME, a parameter-efficient fine-tuning method blending spatial and Fourier adapters through learnable fractional orders.
• It leverages continuous interpolation via a discrete fractional-Fourier transform and chirp–FFT surrogate, ensuring compact updates with reduced interference.
• Empirical results demonstrate FRAME's improvements across commonsense, math, code, and knowledge tasks while maintaining a minimal active parameter budget.

Fractional-Fourier Mixture of Experts, introduced as FRAME in "FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts" (Saliencro et al., 30 Jun 2026), is a parameter-efficient fine-tuning method in which the adaptation domain is itself learned rather than fixed. The method starts from the observation that conventional PEFT schemes reparameterize weight updates in a predetermined basis: low-rank adapters such as LoRA operate in the spatial domain, while spectral methods operate in a fixed Fourier domain. FRAME replaces that fixed design choice with a mixture-of-experts adapter whose experts each carry a learnable fractional-Fourier order, allowing continuous interpolation between the spatial endpoint and the Fourier endpoint. In the formulation reported in the paper, this yields a token-routed adapter family intended to place each low-rank update in the domain where it is most compact, while exploiting the mutual incoherence of different fractional orders to reduce interference and improve multi-task composition (Saliencro et al., 30 Jun 2026).

1. Conceptual setting within PEFT

FRAME is situated at the intersection of low-rank adaptation, spectral parameterization, and mixture-of-experts routing. Its point of departure is the claim that the basis used for expressing an update is a design degree of freedom rather than a fixed architectural constant. In standard LoRA, the update is expressed directly in the spatial coordinates of the hidden state. In Fourier-style adapters, the update is expressed after mapping activations into a spectral basis. FRAME generalizes both by associating each expert with a scalar order $a_i \in [0,1]$, so that $a_i=0$ recovers the spatial domain and $a_i=1$ recovers the Fourier-domain endpoint.

This construction makes the “adaptation domain” a learnable object. The paper’s concise synthesis states that each expert “continuously rotates between the spatial domain ($a=0$, recovering LoRA) and the spectral domain ($a=1$, recovering Fourier/cosine adapters)” (Saliencro et al., 30 Jun 2026). A plausible implication is that the model need not commit globally to one representational basis for all tasks, layers, and token types. Instead, different experts can occupy different positions on a spatial–spectral continuum and be selected per token.

The method is explicitly framed against both spatial and spectral baselines. The abstract reports improvements over strong MoE-LoRA and spectral baselines, including FlyLoRA, FourierMoE, and HMoRA, while keeping the active-parameter budget small. The same source also emphasizes that no single basis is optimal across tasks, layers, or tokens, which is the central hypothesis driving the design.

2. Fractional-Fourier representation and domain interpolation

The mathematical backbone of FRAME is the discrete fractional-Fourier transform (DFrFT) of order $a$, treated as a unitary operator on $d$-dimensional vectors. With $\theta=\frac{\pi}{2}a$, the DFrFT admits a spectral decomposition over real Hermite–Gauss eigenvectors shared across orders:

$$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$

The paper lists several properties used by the method: the identity $\mathcal{F}^{0}=I$, the Fourier limit $a_i=0$0, index additivity $a_i=0$1, unitarity $a_i=0$2, and Parseval preservation $a_i=0$3 (Saliencro et al., 30 Jun 2026). FRAME uses the real-part operator

$a_i=0$4

with $a_i=0$5 and $a_i=0$6, described in the paper as a cosine-type transform. The endpoints therefore coincide with two established adapter domains: the spatial LoRA baseline and a spectral adapter.

A key claim is that distinct fractional orders correspond to distinct phase rotations in the Hermite–Gauss eigenbasis. Proposition 3, labeled “Domain-diversity decorrelation,” gives the bound

$a_i=0$7

with $a_i=0$8 decreasing on $a_i=0$9. In the paper’s interpretation, experts at well-separated orders are decorrelated, reducing interference. This is the theoretical basis for the claim that order diversity assists multi-task composition and adapter merging.

The efficient implementation relies on a chirp–FFT surrogate:

$a_i=1$0

where $a_i=1$1 and $a_i=1$2 are order-dependent complex chirps and $a_i=1$3 is the FFT. The stated procedure is pre-chirp multiplication, FFT, and post-chirp multiplication, with $a_i=1$4 complexity per transform and $a_i=1$5 memory. Proposition 4 states that this surrogate equals $a_i=1$6 up to the discretization error, described in the paper as “up to the $a_i=1$7 sampling error of the discretization,” with an appendix derivation of band-limited discretization error scaling that is reported as near-exact in practice (Saliencro et al., 30 Jun 2026).

3. Adapter construction and mixture-of-experts mechanism

The baseline single-expert LoRA update adds a rank-$a_i=1$8 term to a frozen weight $a_i=1$9:

$a=0$0

with $a=0$1 and $a=0$2. Spatial MoE-LoRA extends this by routing tokens to a top-$a=0$3 subset of experts:

$a=0$4

where $a=0$5 and $a=0$6 returns the active experts.

FRAME replaces the spatial update of each expert with a fractional-domain update. Each expert $a=0$7 has low-rank factors $a=0$8, $a=0$9, and a learnable scalar order $a=1$0:

$a=1$1

Applied to a token $a=1$2, the update becomes

$a=1$3

The special cases are explicit. If $a=1$4, then $a=1$5 and the expert reduces to a spatial LoRA expert. If $a=1$6, then $a=1$7 and the expert becomes a cosine-domain spectral adapter. Intermediate orders yield distinct rank-$a=1$8 subspaces through the rotation $a=1$9.

The full layer is

$a$0

The trainable parameters are $a$1, the orders $a$2, and the router $a$3, while $a$4 remains frozen. In the default configuration, the paper uses $a$5 experts, $a$6 active experts, rank $a$7 per expert, and scaling $a$8. Adapters are inserted at the target modules $a$9 in transformer blocks. Routing is per token, and experts do not share low-rank parameters across orders (Saliencro et al., 30 Jun 2026).

4. Optimization, routing, and implementation details

Orders are optimized as constrained scalars. The paper uses a sigmoid reparameterization,

$d$0

and trains $d$1 by gradient descent with automatic differentiation through the chirp–FFT factorization. The reported order derivative is

$d$2

and the paper states that autodiff through the surrogate suffices in practice. A proposition on bounded order gradients gives

$d$3

together with a corresponding per-token loss-gradient bound involving $d$4, $d$5, $d$6, and $d$7. The stated purpose is training stability.

Routing uses the top-$d$8 softmax router $d$9. To prevent collapse across the spatial–spectral continuum, the method introduces grouped load balancing over order bands. Orders are initialized on a uniform grid in $\theta=\frac{\pi}{2}a$0, experts are partitioned into $\theta=\frac{\pi}{2}a$1 bands, and balancing is applied within each band:

$\theta=\frac{\pi}{2}a$2

Here, $\theta=\frac{\pi}{2}a$3 is band $\theta=\frac{\pi}{2}a$4, $\theta=\frac{\pi}{2}a$5 is the fraction of tokens routed to expert $\theta=\frac{\pi}{2}a$6, and $\theta=\frac{\pi}{2}a$7 is the mean router probability. The paper states that this keeps domain bands utilized while allowing experts to move across bands as orders adapt.

A separate AdamW optimizer is used for the orders, with a smaller learning rate $\theta=\frac{\pi}{2}a$8. The default values are $\theta=\frac{\pi}{2}a$9 and $$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$0, and the total objective is task loss plus $$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$1 with $$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$2. Default training uses AdamW, cosine decay, 3 epochs, bfloat16 precision, effective batch size 128, and sequence length 256 for commonsense or 512 for math, code, and knowledge tasks. Order initialization is a uniform grid on $$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$3, with $$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$4 order bands (Saliencro et al., 30 Jun 2026).

From a systems standpoint, each active expert adds one DFrFT via chirp–FFT plus low-rank maps. The transform cost is $$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$5 per token with $$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$6 memory. The active parameter budget per layer is reported as $$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$7. Relative to spatial MoE-LoRA, FRAME adds $$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$8 scalar orders and $$\mathcal{F}^{a}=\sum_{m=0}^{d-1} e^{-\mathrm{i}m\theta}\,\mathbf{u}_m\mathbf{u}_m^{\top}.$$9 FFTs per token, while keeping low-rank factors identical in size. The paper reports training-time overhead within 7% of spatial MoE-LoRA with the surrogate, whereas the dense transform is 3× slower. The chirps $\mathcal{F}^{0}=I$0 and $\mathcal{F}^{0}=I$1 are recomputed once per step per expert when $\mathcal{F}^{0}=I$2 changes, and the amortized overhead is said to be dominated by the FFT.

5. Empirical results, specialization, and ablations

The principal backbones are LLaMA-3.1-8B and Qwen2.5-7B, with Qwen2.5-1.5B, 3B, and 14B used in a scaling study. Benchmark families are organized as follows: commonsense tasks are BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-easy, ARC-challenge, and OBQA; math tasks are GSM8K, MATH, SVAMP, MAWPS, and AQuA; code tasks are HumanEval and MBPP; knowledge tasks are MMLU and ScienceQA (Saliencro et al., 30 Jun 2026).

On the commonsense suite for LLaMA-3.1-8B, FRAME reports average accuracy 87.1 with active trainable Param% 0.31, compared with 86.3 and 0.39 for the best baseline, identified as MoA. The reported gains are +2.6 over LoRA at 84.5 and +0.8 over MoA. Dataset-level examples include ARC-challenge at 83.4 versus 82.2 for the best baseline and WinoGrande at 87.8 versus 86.9. On Qwen2.5-7B, FRAME reports 88.2 average accuracy at 0.31 Param%, compared with 87.5 at 0.40 for the best baseline; the gains are +2.3 over LoRA at 85.9 and +0.7 over MoA. The paper states improvements across all eight commonsense datasets, including WinoGrande 87.4 versus 86.5 and ARC-challenge 85.0 versus 84.0.

For math, code, and knowledge on LLaMA-3.1-8B, FRAME reports average 49.3, compared with 48.1 for the strongest baselines, listed as FlyLoRA or MoA, and 45.3 for LoRA. The examples given are GSM8K 60.2 versus 59.0, MATH 21.4 versus 20.3, HumanEval pass@1 37.8 versus 36.9, MBPP 42.6 versus 41.5, and MMLU 41.9 versus 40.9.

The interpretability analysis emphasizes learned-order specialization. Experts spread across the $\mathcal{F}^{0}=I$3 continuum, and the spread widens with depth. Order trajectories move away from their grid initialization toward task-appropriate values and stabilize by approximately one-third of training. Token-type routing is reported to be domain-aware: numerals, function words, and content words prefer different orders. This suggests that the learned domain is not merely a layerwise global preference but participates in token-level routing behavior.

Ablations on the LLaMA-3.1-8B commonsense average isolate the contribution of the major ingredients. FRAME full gives 87.1. Fixing $\mathcal{F}^{0}=I$4 yields 85.7, fixing $\mathcal{F}^{0}=I$5 gives 86.0, using a fixed diverse grid rather than learned orders gives 86.4, and a single shared learnable order gives 86.3. Replacing grouped balancing with global balancing gives 86.6; removing balancing gives 85.9. Using a joint optimizer rather than a separate $\mathcal{F}^{0}=I$6 gives 86.5. Replacing the surrogate with exact dense FrFT gives 87.2, only +0.1, but at much higher cost. The paper’s stated takeaway is that per-expert learnable orders and grouped balancing are essential, while the chirp–FFT surrogate is near-lossless (Saliencro et al., 30 Jun 2026).

The efficiency study reports that FRAME sits on the Pareto frontier, with the best accuracy among MoE baselines at modest active cost. Gains persist across backbone scales from 1.5B to 14B, with +0.8 to +1.1 over the best baseline. Sensitivity analysis indicates that accuracy rises with $\mathcal{F}^{0}=I$7 and saturates around $\mathcal{F}^{0}=I$8; $\mathcal{F}^{0}=I$9 gives the best accuracy/cost trade-off, whereas $a_i=0$00 increases cost with only marginal gains. In training-free merging of single-task adapters, FRAME reports the highest accuracy retention, with average retention 87.8%, which the paper treats as consistent with reduced interference from order diversity.

6. Interpretation, misconceptions, and limitations

A common simplification would be to treat FRAME as merely a Fourier-domain adapter with an additional scalar parameter. The formulation does not support that interpretation. The order parameter spans a continuum whose endpoints recover spatial LoRA and a cosine-type spectral adapter, and the paper’s theoretical and empirical arguments depend on the coexistence of multiple experts at different orders rather than on a single spectral representation. The ablations with fixed $a_i=0$01, fixed $a_i=0$02, a fixed diverse grid, and a single shared learnable order all underperform the full model, reinforcing the distinction.

Another possible misconception is that the benefit is exclusively from adding more experts. The reported sensitivity results do show gains with larger $a_i=0$03 up to saturation around $a_i=0$04, but the ablation results indicate that order learning and grouped balancing are central components. Similarly, the paper reports that an exact dense FrFT improves by only +0.1 over the surrogate while being 3× slower, so the practical contribution is not tied to an exact transform implementation.

The limitations are stated explicitly. Overhead grows with the number of active experts $a_i=0$05, and very large $a_i=0$06 erodes efficiency. Each expert uses only a single shared order acting along the input dimension; richer parameterizations such as per-axis or wavelet variants are identified as possible extensions but are not explored. The analyses of learned orders are correlational rather than causal, so the observed specialization patterns do not establish mechanism. As with MoE adapters more broadly, routing collapse can occur; grouped balancing mitigates but does not eliminate it. When a single domain suffices, experts may collapse toward that order, leaving a residual cost that is small but non-zero (Saliencro et al., 30 Jun 2026).

Within the PEFT landscape, FRAME can therefore be understood as a mixture-of-experts adapter in which basis selection is promoted to a learned variable. The paper presents this as a way to unify spatial and spectral parameterizations, exploit fractional-order decorrelation, and preserve the small active-parameter regime characteristic of efficient adaptation.