FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts

Abstract: Parameter-efficient fine-tuning (PEFT) reparameterizes weight updates in a fixed basis: low-rank adapters operate in the spatial domain, while a recent line of spectral methods operates in a fixed Fourier domain. We argue that the choice of domain is itself a design degree of freedom that should be learned, and that no single basis is optimal across tasks, layers, or tokens. We introduce Fractional-Fourier Mixture of Experts, a mixture-of-experts adapter in which every expert carries a learnable fractional-Fourier order that continuously interpolates between the spatial domain (recovering vanilla LoRA) and the Fourier domain (recovering a spectral adapter). Routing tokens through experts that occupy different points on this spatial-spectral continuum lets the model place each low-rank update in the domain where it is most compact, and -- because fractional-Fourier operators of different orders are mutually incoherent -- makes the experts naturally decorrelated, which reduces interference and improves multi-task composition. The order is a single scalar per expert, trained with a separate optimizer, and the transform is computed with an $\mathcal{O}(d\log d)$ chirp--FFT surrogate, so Fractional-Fourier Mixture of Experts adds negligible cost over standard MoE-LoRA. Across commonsense, mathematical, code, and knowledge benchmarks on LLaMA-3.1-8B and Qwen2.5-7B, Fractional-Fourier Mixture of Experts improves over strong MoE-LoRA and spectral baselines -- including FlyLoRA, FourierMoE, and HMoRA -- while keeping the active-parameter budget small, and analysis shows that the learned orders specialize by task and layer in interpretable ways.

• The paper introduces a novel PEFT paradigm that employs learnable fractional-Fourier orders per expert to span adaptive spatial-spectral domains.
• It achieves state-of-the-art accuracy gains with minimal parameter overhead and computational cost across diverse benchmarks.
• Empirical and theoretical insights reveal that dynamic domain adaptation reduces interference and enhances multi-task composability.

FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts

Introduction and Motivation

FRAME introduces a novel parameter-efficient fine-tuning (PEFT) paradigm that generalizes the prevailing spatial (LoRA) and spectral (Fourier) domains used for low-rank adaptation in frozen pre-trained models. Rather than fixing the adaptation domain, FRAME parameterizes it as a learnable fractional-Fourier transform order per expert within a mixture-of-experts (MoE) adapter, allowing adapters to span a continuous spatial-to-spectral spectrum. This approach addresses the empirical observation that the optimal adaptation domain is not universal; it varies by task, model layer, and sometimes input token type. Experimental analysis demonstrates that sweeping the fractional-Fourier order for LoRA-style adapters yields optimal validation accuracy at task-specific intermediate orders rather than at the conventional endpoints.

Figure 1: Optimal adaptation domains depend on task and layer, with intermediate fractional-Fourier orders improving energy compaction and accuracy relative to fixed spatial or spectral domains.

Model Architecture

FRAME extends standard MoE-LoRA by equipping each expert with a learnable fractional-Fourier order $a_i\in[0,1]$, interpolating between the spatial domain ($a=0$) and the Fourier domain ($a=1$). The adaptation update for each expert is constructed as $B_i A_i \mathrm{Re}[\mathcal{F}_{a_i}]$, where $\mathcal{F}_{a_i}$ is the discrete fractional-Fourier transform of order $a_i$. Token-specific routing activates the top-$k$ experts per token, and updates are performed in the corresponding expert's learned domain.

Efficient computation is achieved using a chirp–FFT surrogate for the fractional transform, incurring only $\mathcal{O}(d\log d)$ compute per active expert—comparable to other MoE-PEFT methods. Critically, each expert's order is trained via a separate optimizer with a smaller learning rate to maintain training stability.

Figure 2: FRAME architecture—token routing to top-k experts, each imposing low-rank updates in its own learned fractional-Fourier domain ($a_i$).

Theoretical Properties

FRAME rigorously generalizes both LoRA and spectral adapters: it contains spatial and Fourier MoE-LoRA as strict special cases. The mutual incoherence between experts at well-separated fractional orders provably decorrelates their updates, reducing parameter interference and improving multi-task composability. These theoretical claims are formalized:

• Strict generalization: For any LoRA or spectral expert, a corresponding FRAME expert exists that reproduces its update. Frame's hypothesis class strictly contains both baselines for $N\ge2$ experts.
• Bounded grad stability: Gradients for learnable orders are provably stable, preventing destabilization due to rapid domain change.
• Decorrelation: Expected coherence of updates across experts falls monotonically with order spacing, so domain diversity achieves low-interference specialization.
• Faithful, cheap transform: The fractional transform surrogate closely approximates the true transform with negligible computational cost.

Empirical Results

FRAME sets new state-of-the-art results on a comprehensive suite of benchmarks spanning commonsense, mathematical reasoning, code generation, and knowledge tasks across LLaMA-3.1-8B and Qwen2.5-7B (and other backbone scales). Key findings:

• Score improvements: FRAME consistently outperforms all single-domain and MoE baselines, yielding an average accuracy gain of +0.8 to +2.6 points depending on the backbone and task family, while requiring as little as 0.31% of the full model's parameters.
• Task robustness: Largest gains are observed in more compositional or challenging settings (e.g., ARC-challenge, WinoGrande).
• Efficiency: Surrogate transform keeps wall-clock overhead within 7% of MoE-LoRA, and FRAME lies on the Pareto frontier for accuracy vs. active parameter count.

  Figure 3: FRAME achieves state-of-the-art accuracy at modest active-parameter cost with minimal compute overhead. Gains are consistent across backbone scales and expert configurations.

Mechanistic Insights

Ablation studies reveal that both per-expert learnable orders and grouped load balancing are crucial: fixing the adaptation domain to spatial, spectral, or a static intermediate grid consistently degrades performance. In contrast, per-expert learnable orders induce structured, interpretable specialization across tasks, layers, and even token types:

• Layer specialization: Early transformer layers favor spatial domains; deeper layers migrate towards more spectral orders.
• Token specialization: Numerals, function words, and content words are differentially routed to different domains, evidence for learned token-dependent domain adaptation.
• Expert decorrelation: As predicted theoretically, inter-expert coherence declines with order spacing, boosting overall specialization and adaptability.

  Figure 4: Learned orders spread across the spatial–spectral continuum, specialize by layer depth, and adapt dynamically over the course of training.

  

  Figure 5: Inter-expert coherence drops with increasing order spacing, demonstrating effective expert decorrelation through domain diversity. Grouped balancing maintains domain utilization.

Implications and Future Developments

The principal implication is that adaptation domain itself is a critical, learnable axis for model specialization, composition, and interference reduction in PEFT. FRAME's design enables fine-grained, parameter-efficient control over how information is encoded and adapted, promising greater robustness and utility for multi-task or continual learning, adapter merging, and potentially other components beyond adapters (e.g., in full fine-tuning or attention mechanisms). The empirical and theoretical results underscore the advantage of trainable, non-Euclidean domains for parameter-efficient transfer.

Future work may extend this paradigm to richer, possibly per-axis or multi-dimensional domain parameterizations (e.g., anisotropic or wavelet-style transforms), or combine learnable projections with additional methods for expert decorrelation and modularity. Applying FRAME-style adaptation domains to complex merging, continual learning, or hierarchical adaptation scenarios are particularly promising avenues.

Conclusion

FRAME reframes the adaptation domain in parameter-efficient fine-tuning as a learnable, per-expert attribute rather than a fixed prior. By letting experts operate at arbitrary fractional-Fourier orders, it generalizes and subsumes spatial and spectral PEFT approaches, producing higher-capacity, better-specialized, and less-interfering adaptation at negligible added computational cost. FRAME’s results strongly suggest that the adaptation domain should become a first-class hyperparameter in the development of future PEFT, multi-task, and model composition methodologies.