CompAs: Compositional Adapter Synthesis

• CompAs is a meta-learning framework that generates LoRA-format adapters from context chunks, enabling scalable and efficient LLM inference.
• It employs a two-stage pipeline with a generator network and a student model to emulate prompt concatenation through additive adapter composition.
• Empirical results demonstrate improved accuracy, reduced inference costs, and robustness compared to traditional in-context learning techniques.

CompAs

CompAs denotes a meta-learning framework for compositional adapter generation in LLMs, introduced by Jukić, Lin, and Šnajder in "Context Parametrization with Compositional Adapters" (Jukić et al., 26 Sep 2025). It provides a differentiable mechanism for mapping input context—such as instructions, demonstrations, or retrieved passages—into low-rank adapters whose linear combination emulates standard prompt concatenation. The design addresses context-window inefficiencies, inefficacy and instability of conventional in-context learning (ICL), and the inflexibility of supervised fine-tuning, supporting scalable and robust LLM deployment.

1. Architectural and Methodological Foundations

CompAs operates as a two-stage, meta-learned pipeline. A generator network, $G$, reads each context chunk $\mathbf{c}_i$ and emits a LoRA-format adapter $\boldsymbol{\phi}_i$ parameterizing a subspace in the adapter space $\Phi \subset \mathbb{R}^m$. These adapters are independently generated and composed additively: $$\boldsymbol{\phi}_{\text{comb}} = \sum_{i=1}^{k} \boldsymbol{\phi}_i.$$ A frozen "student" LLM then answers queries $\mathbf{q}$ with the joint adapter loaded, mimicking the effect of concatenating all context segments before inference. Distillation occurs by matching the logits of the adapter-equipped student with those of a teacher LLM exposed to token-level concatenation.

The generator $G$ is architected around a shared frozen backbone complemented by a lightweight contextual encoder and a linear bottleneck, producing LoRA adapter modules per transformer layer. The LoRA mechanism ensures efficient, low-parameter updates, while the linear projection and absence of nonlinearity between context encoding and adapter vectors enforce compositionality.

2. Mathematical Formulation and Training Objectives

For each context chunk $\mathbf{c}_i$, $G(\mathbf{c}_i) \to \boldsymbol{\phi}_i$. The student model with cumulative adapter $\Phi = \sum_i \boldsymbol{\phi}_i$ processes only the query sequence $\mathbf{c}_i$0, while the teacher receives the entire [context; query] prompt. Three key objectives are optimized:

1. Distillation loss: KL divergence between student (adapter) and teacher logits, over both single- and multi-chunk settings,

$\mathbf{c}_i$1

2. Additivity regularizer: Encourages generator linearity by matching the sum of single-chunk adapters to the adapter generated from concatenated context,

$\mathbf{c}_i$2

with $\mathbf{c}_i$3 the LoRA matrices per module and normalization/statistical stabilization parameter $\mathbf{c}_i$4.

3. Reconstruction (invertibility) loss: A special reconstruction token is prepended, and the student is trained to autoregressively decode $\mathbf{c}_i$5 given $\mathbf{c}_i$6, enforcing that adapters retain full context information:

$\mathbf{c}_i$7

Total loss is a weighted sum $\mathbf{c}_i$8.

3. Composable Adapter Merging and Context Recovery

A core property of CompAs is that addition in adapter space approximates concatenation in prompt space: for context blocks $\mathbf{c}_i$9,

$\boldsymbol{\phi}_i$0

Adapters consist of low-rank pairs $\boldsymbol{\phi}_i$1 per layer, summed module-wise for merging. No extra nonlinearity or combinatorial layer is introduced, yielding a monoid-homomorphic structure.

Reversibility is achieved: given a merged adapter, prompting the student with $\boldsymbol{\phi}_i$2 allows accurate reconstruction of input context, verified via token-level $\boldsymbol{\phi}_i$3 scores (high-80s to low-90s maintained for up to 3 chunks summed) (Jukić et al., 26 Sep 2025). This property enables both safety (adapter auditing) and theoretical frameworks for context inversion.

4. Comparison with In-Context Learning and Generator-Based Alternatives

Standard ICL requires re-encoding the prompt for every inference, yielding hard context limits and quadratic scaling with context length. CompAs amortizes context encoding: each chunk is processed once to produce an adapter. Subsequent queries involving the same context incur inference costs independent of total $\boldsymbol{\phi}_i$4: $\boldsymbol{\phi}_i$5 Empirical evaluation shows up to $\boldsymbol{\phi}_i$6 speedup and 20% lower peak memory at 16-shot scale, with superior accuracy and variance reduction relative to ICL and generator hypernetwork methods (e.g., GenAda). Compositionality failures and instability present in non-linear hypernetwork generators are minimized. CompAs matches or exceeds fine-tuned per-context adapters (WILDA) in 11 of 12 high-shot tasks, but with greater flexibility since no retraining is required per chunk (Jukić et al., 26 Sep 2025).

5. Empirical Results and Theoretical Guarantees

CompAs is validated on multiple-choice and extractive QA tasks: MMLU (59 subjects), ARC-Challenge, HotpotQA. With increasing context length (shots or supporting paragraphs), CompAs maintains or improves performance where baseline ICL and generator-based approaches degrade. At 16-shots, CompAs delivers a 2–6 point accuracy gain (absolute), up to +4 EM and +2–4 F1 (HotpotQA), and reduces inference costs by a factor of four (Jukić et al., 26 Sep 2025).

Theoretical bounds (Theorem 1) relate divergence between student and teacher logits under compositional adapter addition to generator additivity error $\boldsymbol{\phi}_i$7 and student–teacher alignment error $\boldsymbol{\phi}_i$8: $\boldsymbol{\phi}_i$9 extending linearly in $\Phi \subset \mathbb{R}^m$0. This underpins the empirical observation of stable accuracy as context scales.

6. Applications, Limitations, and Extensions

CompAs is beneficial in three primary regimes:

1. Overlong prompts: Contexts exceeding the LM’s window can be chunked, encoded as adapters, and summed, sidestepping hard window limits.
2. Efficient batch LLM serving: Repeated queries on shared context amortize cost, crucial for retrieval-augmented pipelines, multi-task orchestration, or interactive applications.
3. Safety and control: Reversible adapter encoding enables extraction, verification, or repudiation of encoded context.

Limitations include non-exact linearity in generator (small but finite $\Phi \subset \mathbb{R}^m$1), and that errors can compound linearly with the number of composed chunks. Extension areas include handling chunk ordering, scaling additivity to even higher $\Phi \subset \mathbb{R}^m$2, and incorporating richer compression or attention mechanisms within the generator for extreme prompt lengths.

7. Related Work and Context

CompAs builds upon the LoRA adapter paradigm, prior adapter generation schemes (e.g., context-conditioned hypernetworks), and literature on efficient parameter-efficient fine-tuning. Its algebraic composition property contrasts with both vanilla ICL and explicit multi-step prompt concatenation, and it establishes a new regime for task-conditioned, context-parametric inference in LLMs (Jukić et al., 26 Sep 2025). Empirical and theoretical validation position it as the most compositional, efficient, and robust context parametrization technique among published meta-learning adapter frameworks as of its introduction.