Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
This presentation explores a novel architecture that generates task-specific neural network weights on-the-fly from live interaction data rather than selecting from a fixed bank of stored experts. We examine how this approach compiles evidence into low-rank weight modifications, maintains a persistent belief state across conversation turns, and addresses the fundamental challenge of incorporating runtime information into deployed language models without retraining.Script
Deployed language models remain frozen after training, unable to incorporate the facts, corrections, and demonstrations users provide during live interaction. This paper asks a simple but radical question: what if a model could compile runtime information directly into temporary weight configurations that persist and evolve across conversation turns?
The architecture separates a frozen shared base network from generated low-rank updates. For each interaction, a hypernetwork produces adapter weights from a compact latent code, creating effective configurations without storing a massive expert bank. The key departure is that these codes can change during a session, maintained through online Bayesian belief updates rather than being computed once and frozen.
Experiments reveal a striking structure dependence. On SQuAD with short clean passages, in-context prompting crushes data-to-weights: 85 versus 52 F1. But the relationship completely reverses on long, noisy, multi-hop datasets, where compiled weights reach 48 F1 compared to just 34 for prompting on MS MARCO. Compiling evidence into weights only wins when prompt length, distractor noise, or repeated reuse penalizes the context window.
A single fixed-size code cannot scale. Even with oracle passage ordering and no truncation, performance drops from 51.6 to 46.9 F1 as the evidence pool grows from 8 to 32 passages. The code saturates, losing representational fidelity. This motivated a shift to multiple bounded per-passage codes with learned selection, avoiding the compilation bottleneck entirely.
The trained activation-based selector consistently outperforms dense retrieval by 8 to 12 points in routing accuracy, reaching 53 percent on MS MARCO and 70 percent on 2WikiMultihopQA. Late-layer activations concentrate the discriminative signal, allowing the model to identify which compiled code contains the answer-bearing information. As evidence pools grow from 8 to 64 passages, single-code compilation collapses to 28 F1 while routed selection holds steady near 48.
The persistent belief accumulates evidence across turns at fixed computational cost, sharpening its routing accuracy as conversations establish their topic, while memoryless methods remain flat and concatenated-history retrieval eventually degrades under its own growing query length. The work reframes generated weights not as a one-shot compilation trick, but as a session-persistent adaptation state that learns what matters as you talk. To explore this architecture further and create your own videos on emerging research, visit EmergentMind.com.
