Learn-to-learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM

Abstract: Conventional LLMs may suffer from corpus heterogeneity and subtle condition changes. While finetuning can create the catastrophe forgetting issue, application of meta-learning on LLMs is also limited due to its complexity and scalability. In this paper, we activate the meta-signal of $β$ within the SwiGLU blocks, resulting in a meta-gating mechanism that adaptively adjusts the nonlinearity of FFN. A hypernetwork is employed which dynamically produces $β$ on textual conditions, providing meta-controllability on LLMs. By testing on different condition types such as task, domain, persona, and style, our method outperforms finetuning and meta-learning baselines, and can generalize reasonably on unseen tasks, condition types, or instructions. Our code can be found in https://github.com/AaronJi/MeGan.

• The paper introduces MeGan, a novel meta-gating mechanism that uses hypernetworks to adapt large language models to arbitrary textual conditions without retraining the backbone.
• It leverages neuro-inspired gain control and β-SwiGLU activations to achieve parameter-efficient, context-sensitive modulation across various tasks and domains.
• Empirical results show MeGan outperforms conventional methods with significant gains in ROUGE-L, BLEU-2, and accuracy across multiple benchmarks.

Hypernetwork-Driven Meta-Gating for Arbitrary Textual Conditioning in LLMs

Motivation and Neuro-Inspired Paradigm

The paper "Learn-to-learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM" (2605.01973) addresses the inherent limitations of LLMs in dynamic adaptation to nuanced textual conditions (e.g. task, domain, persona, style, sentiment). Standard finetuning yields catastrophic forgetting and alignment tax, whereby adaptation to new contexts compromises global capabilities. The authors propose MeGan, a paradigm modeled after neuroplasticity with neuromodulator-driven gain control: a hypernetwork computes meta-signals from condition inputs to steer the gating of FFN activations in the LLM backbone.

Figure 1: The MeGan paradigm, with a hypernetwork meta-controlling LLM FFN gating, analogous to neural systems with neuromodulator-driven synaptic meta-plasticity.

This neuro-inspired intervention replaces static self-gating with meta-gating, providing context-sensitive modulation of the FFN block nonlinearity—a mechanism aligning with the neuronal f-I curve, and empirically linked to semantic encoding in earlier MLP layers.

Meta-Learning Paradigms and Architectural Overview

MeGan is compared to four principal meta-learning paradigms for LLMs: adapter-based PEFT, meta-in-context learning, LoRA-based adaptation, and text-driven hypernetwork architectures. Unlike text-to-LoRA hypernetworks—which generate LoRA weights from task descriptions—MeGan produces meta-gating signals (parameter $\beta$) per layer, enabling fine-grained modulation in $\beta$-SwiGLU FFNs without altering the backbone parameters.

Figure 2: Comparative analysis of LLM meta-learning paradigms, highlighting the distinctiveness of MeGan’s hypernetwork-driven meta-gating architecture.

The core MeGan structure consists of:

• Backpropagation-frozen LLM backbone (e.g. Llama-3.1-8B-Instruct)
• A lightweight hypernetwork leveraging cross-attention and layer embeddings to generate layerwise $\beta$ for $\beta$-SwiGLU
• Information compression via reduced hidden dimension ($R$), yielding parameter efficiency and key semantic extraction

  Figure 3: Detailed MeGan architecture, with textual condition $z$ processed by the hypernetwork to produce layerwise $\beta$ for meta-gating.

Methodological Details

MeGan formalizes meta-learning over arbitrary textual conditions $z$, thus broadening the scope beyond task-specific meta-learning. Data triples $(x, y, z)$ are constructed, allowing MeGan to adaptively generate outputs conditioned on $z$ (e.g., style, persona, domain) at inference.

The $\beta$0-SwiGLU FFN block generalizes SiLU activation to $\beta$1, where the hypernetwork projects $\beta$2 and layer index $\beta$3 into $\beta$4 using cross-attention and bias-free MLP. The information bottleneck reduces $\beta$5 then projects to $\beta$6, enforcing compression and modularity in control signals.

The training objective is a linear combination of cross-entropy (SFT) and $\beta$7 regularization on $\beta$8 to constrain deviation, ensuring stable training dynamics.

Experimental Validation

MeGan is evaluated on diverse datasets spanning persona (Persona-Chat), domain (AdaptSum), task (CrossFit/UnifiedQA, SNI), style (GYAFC, MIC), and sentiment (SST, Amazon, IMDB). Baselines include ICL, CoT, Plan-and-Solve, Metacognitive Prompting, LoRA, SFT, meta-in-context approaches, meta-LoRA, and Text-to-LoRA. MeGan consistently outperforms all baselines on adaptation metrics (ROUGE-L, BLEU-2, accuracy), both in low-resource and unseen tasks:

• Persona-Chat: MeGan achieves 23.15 R-L, outperforming Text-to-LoRA (21.75) and LoRA (21.35)
• AdaptSum: MeGan reports 41.85 R-L, significantly surpassing MLtD and SFT baselines
• CrossFit/UnifiedQA: Average performance of 72.2% in LR and 70.9% in unseen tasks, well above MetaICL and LoRA
• SNI benchmarks: Highest average accuracy over 10 general benchmarks

Scalability is demonstrated with consistent performance across model sizes on Persona-Chat/GYAFC, validating robustness under parameter constraints.

Figure 4: Model scalability across parameter ranges for low-resource and zero-shot datasets.

Meta-Gating Signal Analysis

Distributional analysis of $\beta$9 across textual conditions reveals strong semantic alignment and adaptive modulation. For sentiment, $\beta$0 centers between $\beta$1 and $\beta$2, indicating nonlinearity encoding consistent with human intuition.

Figure 5: Distribution of $\beta$3 by textual condition: meta-gating dynamically customizes FFN activation slopes.

t-SNE visualization confirms domain-aware grouping of $\beta$4 for AdaptSum and GYAFC, signifying layerwise adaptation to semantic context.

Layerwise Modulation

Layerwise averaging of $\beta$5 demonstrates increasing nonlinearity with depth, correlating with heightened abstract semantic representation—a finding consistent with prior activation patching studies.

Figure 6: Layer-dependent modulation of $\beta$6 for nonlinearity in FFN blocks.

Ablation and Case Analysis

Ablation studies confirm necessity of meta-prompt, layer embedding, and regularization. Removal of any component degrades adaptation performance. Sample cases showcase MeGan’s generation quality and consistency under diverse arbitrarily conditioned prompts.

Practical and Theoretical Implications

MeGan introduces meta-control at the activation-function level, yielding parameter-efficient, context-sensitive adaptation for LLMs without retraining. The neuro-inspired paradigm supports:

• Universal adaptation across arbitrary textual conditions, not constrained to structured task descriptions
• Fine-grained, interpretable modulation via activation nonlinearity, enhancing control over stylistic, domain, persona-based, and affective outputs
• Robustness against catastrophic forgetting and alignment tax

Practically, this enables deployment of LLMs in personalized systems, expert-level emotional support agents, and multi-task dialog settings with swift, controlled adaptation. Theoretically, this underpins advances in meta-plasticity modeling and layered information bottleneck utilization. Future directions may include expanding the meta-gating mechanism to other architectural elements (e.g., attention, recurrent modules), integrating richer semantic embeddings, or exploring cross-domain transfer with compressed hypernetworks.

Conclusion

MeGan advances LLM meta-learning by equipping frozen backbones with a hypernetwork-driven meta-gating mechanism for layerwise activation control. Its parameter-efficient design, robust performance across heterogeneous tasks and conditions, and neuro-inspired architecture positions it as a versatile model for learn-to-learn across arbitrary textual contexts. The paper delivers strong empirical evidence that meta-gating of FFN nonlinearity, modulated by semantic hypernetworks, yields superior generalization and adaptation in LLMs (2605.01973).