Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data

Abstract: The scaling laws hold that a LLM grows more capable with more parameters and more training data, and Mixture-of-Experts (MoE) architectures have ridden these laws to remarkable results, activating only a fraction of an enormous stored parameter bank for each token. That success is built on static pretraining data. A deployed model faces a different world, where much of the data that would make it more useful is not in its training set but in the live interaction it is currently handling, such as the facts a user supplies or the corrections they give. A conventional model cannot learn from this data, because its weights are frozen after training. Instead, the knowledge and behaviour supplied at run time are placed in the prompt, by retrieval or instruction, and re-read on every request only to be discarded once the request ends. We ask how an architecture could learn from live interaction by writing it into its weights. Taking inspiration from MoE, we propose the \textbf{Infinite-Parameter LLM}. A compact hypernetwork turns the data given at run time into a low-rank modulation of a shared base network, so the feed-forward weights are generated from live data rather than stored in a fixed bank. Where prior weight generators read the context once and freeze, we carry a Bayesian belief over the generator's latent code and update it online, so the effective weight is re-derived from that evolving belief as the session proceeds rather than fixed after one read. The stored footprint stays fixed, yet the weights the model can compile are effectively infinite. For the knowledge and behaviour supplied at run time, carrying them in the weights rather than the prompt is amortized in compute, frees the context window, persists across turns, and can generalise better than in-context use. We specify an evaluation protocol that tests exactly this against in-context learning and retrieval.

• The paper presents a new architecture for run-time adaptation in language models, described as `infinite-parameter LLMs,` that generates task-specific weights from live data using a hypernetwork and an online belief state over latent codes.
• Experiments show that while in-context prompting outperforms data-to-weights in short, clean evidence, data-to-weights is superior on long, noisy tasks and multi-hop problems.
• A categorical routing mechanism enhances selection accuracy over precompiled generated codes, outperforming BM25 and dense retrieval methods.

The paper proposes an inference-time adaptation architecture in which a LLM generates task- and interaction-specific FFN weights from live data rather than selecting experts from a fixed stored bank. Its central claim is deliberately narrower than the name “infinite-parameter LLM” might suggest: the resident parameter count remains finite, and the model’s knowledge capacity is not unbounded; however, the set of effective weights that can be instantiated from continuously generated latent codes is not restricted to a finite collection of stored experts. The proposal combines three components: a shared base FFN, a hypernetwork that produces low-rank weight modifications from a latent code, and an online belief state over those codes. The work is presented as an overview of conditional computation, hypernetworks, parameter-efficient adaptation, Bayesian filtering, and continual inference rather than as a new isolated primitive (2609.18842).

Motivation and conceptual framing

The paper begins from the distinction between static training data and live interaction data. Scaling-law results establish that model capability depends systematically on data, parameters, and compute (Kaplan et al., 2020, Hoffmann et al., 2022). Yet deployed models remain frozen after training. Facts supplied by users, corrections, demonstrations, task instructions, and agent outcomes can affect the current prompt or an external memory system, but normally cannot alter the model’s weights. In-context learning and retrieval therefore repeatedly reintroduce the same information at inference time and discard it after the request or turn.

The proposed architecture treats this as a mismatch between the source of useful data and the mechanism by which models acquire capability. A deployed model receives potentially valuable data during use, but its parameters cannot incorporate that data without an explicit adaptation mechanism. The paper therefore asks whether run-time information can be compiled into a temporary, low-dimensional weight configuration that persists across turns and evolves as additional evidence arrives.

Mixture-of-Experts provides the conceptual starting point. In a conventional MoE, the effective FFN weights vary with the token through a gate over a stored expert bank:

$W_{\mathrm{eff}}(x) = \sum_i g_i(x)W_i.$

Only a small subset of experts is activated per token, but the entire bank must remain available. This creates a separation between computation and storage: sparse activation reduces token-level computation, whereas the model still stores a large fixed collection of experts. The paper retains the dynamic-weight perspective while rejecting the fixed-bank assumption. The effective expert is generated on demand from a compact latent representation, and that representation is allowed to change during a session.

This framing distinguishes the proposal from bank-free but frozen methods such as $\mu$MoE and $\infty$-MoE, which avoid explicitly materializing large expert banks but still derive their behavior from fixed parameterizations [(Wallis et al., 2024); 2609.????]. It also distinguishes the work from one-shot hypernetwork methods such as Text-to-LoRA and SHINE, which read context once, produce an adapter, and hold it fixed for the turn [2502.????; 2606.????]. The paper’s claimed contribution is specifically the coupling of generated weights with persistent online inference over the generating latent code.

Architecture: generated FFN experts over a shared base

The model modifies only selected FFN sublayers of a decoder-only Transformer; attention and the sequence-mixing path remain unchanged. Each modified layer contains a frozen shared base FFN and a generated low-rank additive update. For a base projection $W_0$, the effective projection is

$W(z) = W_0 + \Delta W(z),$

where the update is factorized as

$\Delta W(z) = B(z)A(z)^\top.$

The factors have rank $r \ll d$, so the update is applied without materializing a full dense matrix. The additional computation scales with $r(d+h)$ rather than with the full FFN cost $hd$, making the adaptation comparatively inexpensive when the rank is small. The evaluated configuration uses a SwiGLU base, rank $r=8$, and latent dimension $\mu$0. Generated updates are applied to the gate, up, and down projections of selected FFN layers.

This structure provides a clear separation between stored and effective parameters. The model stores the base network, the encoder-hypernetwork, the code-to-LoRA mapping, and the selector or inference machinery. It does not store an expert bank. For a given code, the model can nevertheless instantiate a distinct effective FFN. The resulting family is continuous in the framework’s most general formulation, although the experiments use a categorical approximation over a finite working pool of materialized codes.

The paper is careful to restrict the interpretation of this construction. Generating weights does not increase the information capacity of the resident network beyond what its parameters can encode. The “infinite” designation refers to the reachable family of effective configurations, not to an infinite knowledge store. If no relevant information is supplied at inference time, the model remains limited by the knowledge and behavior encoded in its base and generator.

Belief-based online adaptation

The key methodological distinction is that the latent code is not necessarily computed once and frozen. The model maintains a belief over possible codes and updates that belief as new evidence arrives. In the general formulation, the belief is a distribution $\mu$1 over a continuous or discrete latent space. Given observations accumulated over a time interval, the update follows recursive Bayesian filtering:

$\mu$2

The observation likelihood may be derived from the model’s own autoregressive likelihood or from explicit feedback, including scalar rewards and preference comparisons. Exact inference would require evaluating the likelihood and updating the latent posterior, potentially involving backward computation. The proposed deployment mechanism therefore amortizes the filter with a forward recognition network trained to approximate the recursive posterior.

The paper identifies three update cadences. The contextual variant has no explicit persistent belief and resembles ordinary in-context inference. The session variant updates the belief once per turn, preserving a task-specific state between turns. The fast-filter variant updates the belief per token through a low-dimensional side state. All three use the same conceptual machinery, differing only in the observation window and update frequency.

The Bayesian formulation is intended to provide three advantages over point-estimate test-time training. First, posterior dispersion supplies an uncertainty signal. Second, precision-weighted updates are supposed to implement a stability–plasticity trade-off: uncertain latent directions remain adaptable, whereas confident directions are protected. Third, process noise can reopen plasticity when the interaction distribution shifts. These claims are theoretically plausible within the proposed filtering formulation, but the paper does not experimentally validate the continuous posterior variant or demonstrate calibrated uncertainty. In the evaluated system, the belief is categorical and the online mechanism is effectively a learned selector over materialized code atoms.

Evaluated categorical instantiation

The experiments instantiate the belief as a categorical distribution over codes $\mu$3, where each code is generated from a knowledge unit by the data-to-weights encoder. The belief is

$\mu$4

and the recursive update is

$\mu$5

In practice, a selector scores the current layer activation against learned keys associated with the codes. The implementation uses top-1 selection, so one generated expert is applied at each layer rather than a mixture of several generated experts.

The generator is adapted from the SHINE-style pipeline. Evidence is processed together with learnable memory tokens, whose hidden states are collected across layers. A memory-to-parameter network maps these representations to a latent code that is reshaped into LoRA factors for the base FFN. Codes can be computed once per knowledge unit and cached. At generation time, the model does not reread the original evidence; it performs code selection and applies the corresponding factored low-rank update.

This design has an important limitation relative to the paper’s general framework. A categorical belief can select among existing materialized codes but cannot smoothly move to a code outside the current pool. If the relevant behavior or knowledge is not represented by one of the available atoms, adaptation requires materializing another code. The continuous-Gaussian alternative, which would infer a posterior over offsets in latent space, is described but not evaluated.

Data-to-weights versus in-context prompting

The first experimental question is whether supplied evidence can be used effectively after being compiled into generated weights and removed from the prompt. The evaluation uses SQuAD, HotpotQA, 2WikiMultihopQA, MuSiQue, and MS MARCO v2.1. These datasets vary from short, clean single-passage evidence to long, noisy, multi-passage and multi-hop evidence.

The central result is a strong dependence on evidence structure:

Dataset	Closed-book F1	In-context F1	Data-to-weights F1
SQuAD	20.2	85.3	51.8
HotpotQA	22.1	58.7	60.4
2WikiMultihopQA	24.5	55.5	58.1
MuSiQue	15.2	40.9	45.3
MS MARCO v2.1	16.8	33.6	48.0

The result contradicts any universal claim that generated weights are superior to prompting. On SQuAD, where the evidence is short, clean, and used once, in-context prompting substantially outperforms data-to-weights: 85.3 versus 51.8 F1. The prompt is the appropriate mechanism in this regime because the model can directly attend to the evidence without paying a compilation cost.

The relationship reverses for longer and noisier evidence. On MS MARCO, data-to-weights reaches 48.0 F1 compared with 33.6 for in-context prompting. The advantage is also present on the multi-hop datasets, although the margins are smaller. The implication is specific: compiling evidence into weights is most useful when the prompt representation is penalized by length, distractors, repeated reuse, or multi-hop integration. It is not a general replacement for in-context learning.

Dilution, truncation, and the need for code selection

The paper’s dilution study examines how a fixed-size generated code behaves as additional distractor passages are included. Even under oracle ordering, where the answer-bearing passage is placed first and no truncation occurs, performance declines as the evidence pool grows. At a 3,000-token encoder budget, F1 falls from 51.6 to 48.6 and then 46.9 as the pool grows from 8 to 16 and 32 distractor passages.

This establishes that code saturation is not reducible to prompt truncation. A fixed-size code loses representational fidelity when asked to encode too much evidence, even when all passages fit in the encoder window. With shuffled passage order, performance declines further. At a 1,300-token budget, the answer passage is often truncated; at 3,000 tokens and up to 32 passages, truncation disappears, but a residual gap of approximately 5 F1 remains because buried answer passages are encoded less faithfully than fronted passages.

The paper reports an oracle–realistic gap of approximately 8–20 F1 depending on context budget and pool size. This result motivates a shift from one large evidence compilation to multiple bounded compilations followed by selection. The argument is architectural rather than merely empirical: one-shot compilation is limited by both the capacity of the code and the encoder’s inability to know which passage deserves foregrounding, whereas per-passage codes keep each encoding within a bounded regime and defer evidence identification to a selector.

Routing over generated codes

The routing experiment evaluates whether a learned selector can identify the code corresponding to the answer-bearing passage. The baselines include random selection, BM25, dense retrieval with bge-small, a zero-shot confidence heuristic, and an oracle that scores each code using the likelihood of the true answer.

The oracle results show that the generated codes are separable: top-1 accuracy is 78.7 on MS MARCO, 80.9 on HotpotQA, 82.8 on 2WikiMultihopQA, and 70.1 on MuSiQue. However, zero-shot code confidence performs poorly, reaching only 22.7, 24.0, 23.8, and 20.4 respectively. Thus, the information needed for routing exists in the code representations, but it is not automatically recoverable by a naive confidence score.

The trained activation-routed selector outperforms dense retrieval on every dataset:

Router	MS MARCO	HotpotQA	2Wiki	MuSiQue
Random	10.0	10.1	12.3	10.3
Zero-shot confidence	22.7	24.0	23.8	20.4
BM25	20.7	30.5	34.3	22.5
Dense retrieval	45.3	52.2	58.1	40.9
Trained selector	53.3	62.1	70.1	53.0

The margins are 8.0 points on MS MARCO, 9.9 points on HotpotQA, 12.0 points on 2WikiMultihopQA, and 12.1 points on MuSiQue. These are strong results for the routing task, although they should not be conflated with a universal improvement in answer generation: the experiment measures identification of a relevant code, not independent end-to-end performance under all retrieval and prompting conditions.

The ablations indicate that routing information is concentrated in later model layers. Early-layer activations are close to random, whereas late-layer summaries provide most of the discriminative signal. A single late-layer summary outperforms pooling all memory tokens, suggesting that the selector benefits from a representation already specialized to the query rather than from a generic aggregation of encoded evidence.

The end-to-end pool-size experiment further supports selection over one-shot compilation. As the pool grows from 8 to 64 passages, the single large read declines from 48.8 to 27.8 F1. The selector remains nearly constant, moving from 48.1 to 47.6 F1. The implication is that the proposed decomposition—bounded per-item compilation plus query-time selection—avoids the degradation caused by forcing a single code to represent an increasingly large and heterogeneous pool.

Cross-turn accumulation

The final experiment evaluates whether the belief state improves routing over a multi-turn interaction. Conversations begin with an explicit topic-setting turn and then include both self-contained questions and context-dependent follow-ups. The accumulation analysis is restricted to context-dependent turns, for which the current question alone is insufficient.

The persistent categorical belief is updated token by token, with a forgetting parameter controlling the degree to which previous evidence is retained. It is compared with per-question retrieval, a memoryless selector, and retrieval over the concatenated conversation history. The reported qualitative pattern is that the accumulated belief improves as the conversation establishes its topic, while the memoryless methods remain flat and fail on ambiguous follow-ups. Concatenated-history retrieval initially improves but later degrades as the query grows and becomes diluted.

The computational comparison is central. Concatenated-history retrieval incurs a per-turn cost that grows with accumulated history, whereas the belief update maintains a fixed cost of $\mu$6 code comparisons per layer. The paper therefore claims a joint advantage in which the persistent belief becomes more accurate over time while retaining flat per-turn routing cost.

This result supports the paper’s principal distinction from one-shot generators: the generated weights are not only a compressed representation of a single context, but part of a persistent session state. However, the evidence is limited to authored conversations constructed from the benchmark datasets and to categorical top-1 routing. It does not yet establish performance on naturally occurring long-horizon interactions or on settings where the relevant information is distributed across multiple codes rather than represented by one selected atom.

Limitations and open questions

The most important limitation is that the experiments do not evaluate the paper’s most technically ambitious Bayesian construction. The continuous latent-code posterior, uncertainty-calibrated precision, process-noise-controlled forgetting, and smooth code-to-weight mapping are proposed but left for future work. The implemented system uses a categorical pool of materialized codes and top-1 selection. Consequently, its adaptation behavior is closer to persistent routing over generated adapters than to fully continuous Bayesian weight adaptation.

The approach also depends on supplied evidence. It does not enlarge closed-book knowledge capacity and cannot compensate for absent or incorrectly encoded information. The strong SQuAD result for in-context prompting demonstrates that compilation into weights is not intrinsically better. Its advantage is conditional on evidence length, noise, repeated use, and interaction horizon.

The data-to-weights generator can saturate, truncate, or underrepresent buried evidence. The code pool addresses these problems by decomposing evidence into bounded units, but it introduces dependence on the quality and granularity of the materialized codes. A code pool that omits the relevant unit cannot be corrected by the categorical selector without generating a new code. Top-1 routing also collapses potentially multimodal posterior beliefs and is poorly matched to answers requiring coordinated use of several passages.

Several empirical claims require broader validation. The experiments rely on held-out benchmark groups and authored multi-turn conversations; they do not test arbitrary user-generated sessions, distribution shifts in knowledge and behavior, or long-term retention under conflicting evidence. The paper mentions selector coverage regularization, but does not report extensive analysis of code concentration or routing collapse. Likewise, although the method is motivated partly by calibrated uncertainty and stability–plasticity control, no calibration, posterior recovery, amortization-gap, or catastrophic-forgetting evaluation is provided for the evaluated categorical model.

Finally, the comparison with retrieval is task-specific. The selector operates over precompiled codes, while retrieval operates over text passages, so the two systems differ in both representation and computation. The reported routing margins are meaningful, but they do not by themselves establish superiority over optimized retrieval-augmented generation systems with reranking, multi-document synthesis, or answer-aware iterative retrieval. The unresolved technical question is whether a continuous posterior over generated low-rank adaptations can preserve the observed routing gains while supporting genuine within-session modification rather than merely selecting among precomputed atoms.

Conclusion

The paper presents a coherent architecture for moving run-time information from the prompt into generated low-rank FFN weights. Its substantive contribution is the integration of a shared-base hypernetwork with an online belief over latent adapter codes, extending one-shot data-to-weights methods toward persistent session adaptation. The experiments show that data-to-weights is inferior to prompting for short, clean evidence but superior on several long, noisy, and multi-hop settings; that a trained selector over compiled codes outperforms BM25 and dense retrieval by 8–12 points in routing accuracy; and that persistent belief accumulation can improve cross-turn routing without increasing per-turn selection cost. The strongest unresolved issue is whether the proposed continuous Bayesian adaptation mechanism, rather than the evaluated categorical code selector, can deliver the claimed uncertainty-aware and genuinely adaptive behavior under open-ended interaction.