On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters

Abstract: Parameter-efficient fine-tuning (PEFT) is usually treated as a cheaper alternative to full fine-tuning. We study a broader role: small trainable adapters as persistent local state on top of strong shared foundation models. In this framing, the base model provides shared competence while adapters carry instance-specific behavior such as preferences, skills, tool habits, and memory-like updates. We organize the problem around three scaling axes: Scale Up, where stronger shared priors make small local updates more useful; Scale Down, where we study how small adapters can be while remaining reliable; and Scale Out, where many persistent adapted instances coexist. MinT provides one infrastructure example for managing adapter identity, revision, provenance, evaluation, and serving residency. Together, the results suggest that PEFT can be a compact substrate for persistent personal models rather than only a budget substitute for full fine-tuning.

• The paper demonstrates that integrating Scale Up, Scale Down, and Scale Out enables persistent personalization of trillion-parameter models using LoRA and RL fine-tuning.
• Empirical results reveal that moderate LoRA ranks and RL-based optimizations reduce training-inference mismatch and ensure stable, high-reward performance.
• The proposed MinT infrastructure orchestrates rapid adapter deployment, supporting diverse, population-level personalization for scalable model adaptation.

PEFT Scaling for Personal Models: From Trillion-Parameter Foundations to Million-Personalized Instances

Introduction and Motivation

This paper reframes Parameter-Efficient Fine-Tuning (PEFT)—particularly variants such as LoRA—not as a mere cost-saving surrogate for full-model adaptation, but as the principal architectural enabler for scaling persistent, personal models atop trillion-parameter foundation models. The authors rigorously decompose the PEFT scaling landscape along three interdependent axes: Scale Up (leveraging stronger shared priors), Scale Down (tightening the marginal cost and instability of adaptation), and Scale Out (orchestrating population-level collections of adapted models). The central thesis is that robust personalization at scale arises only when these axes are reinforced and co-designed, not when treated in isolation.

Figure 1: The three scaling axes of PEFT underpin the transition from shared foundation models to large populations of individual persistent personal models.

Scale Up: Adapting Trillion-Parameter Priors with PEFT

The utility of a personal adapter is fundamentally upper-bounded by the expressivity of the frozen base model. Reinforcement learning (RL) fine-tuning on substantial priors surfaces behaviors that were latent yet unsupported in smaller models; RL is thus prior-limited in the sense that novel, high-reward behaviors must have non-negligible support from the initial policy. LoRA further shifts the optimization landscape by constraining the adaptation to a low-dimensional subspace, dramatically reducing the memory and compute required while preserving access to the high-capacity prior.

Empirical results show that under equal RL budgets, small-rank LoRA adapters on large models (e.g., Qwen-32B, LoRA $r=8$) can outperform full-parameter RL on smaller models (e.g., Qwen-1.5B) in both normalized gain and generalization. The methodology is operationalized at scale: the GRPO LoRA framework demonstrates stable RL reward and task curves for a trillion-parameter mixture-of-experts (MoE) LLM, contingent on joint design of rollout, training, and parallel inference infrastructure.

Figure 2: GRPO LoRA RL maintains stable reward and task-success on a trillion-parameter MoE reasoning model via harmonized deployment protocols.

A key bottleneck at this scale is training-inference mismatch (TIM), especially acute in sparse MoEs: small numerical differences between training and rollout computation cause routing divergences, breaking locality of policy gradients and destabilizing learning. The paper provides evidence that architectures designed for robust routing replay (e.g., R3) maintain near-zero KL divergence in RL updates and sustain superior critic scores and validation accuracy, while others collapse or degrade.

(Figures 3–6)

Figures 3–6: TIM and RL stability signals; R3 eliminates high-magnitude rollout probability drift and suppresses KL divergence and gradient instability.

Ensuring operational correctness requires tight alignment of adapter semantics, sparse execution pathways, and the backend/serving lifecycle. Failure to harmonize these layers leads to semantic drift that cannot be detected from training loss alone, especially in GLM5/5.1 and MoE settings.

Scale Down: Minimizing and Stabilizing Local Adaptation

The adaptive unit must be compact, robust, and reliably trainable to enable persistent model deployment at population scale. The paper provides extensive ablations mapping LoRA rank to adaptation efficacy. Middle ranks ($r=16$ to $32$) provide the highest expected reward with manageable variance and should be default for deployment. Extremely low ranks ($r=1$ to $4$) are not fundamentally underpowered but exhibit unreliable convergence—high seed dependence and optimization brittle points.

Figure 3: Mean gain by rank—higher mean and lower variance are observed for moderate ranks, while the low-rank regime suffers seed-level instability.

Figure 4: Mean--best separation exposes that low-rank adapters are limited by reliability, not intrinsic capacity; top seeds can match higher-rank performance.

RL-native initializations (OLoRA-tail, geometry-aware minor subspace) resolve much of the seed and batch instability, preventing abrupt collapse and overlarge policy drift implicated by principal component initialization. For instance, on Qwen3-8B, OLoRA-tail (rank 1) gives consistent $+20\%$ $\Delta$Pass@1 gain across batch sizes, compared to LoRA, which degrades with increased batch size and suffers catastrophic instability.

Figure 5: Rank-1 OLoRA-tail adapters maintain consistent gains across batch sizes, while vanilla LoRA performance is unstable and batch-size sensitive.

Efficient transfer of hyperparameters across ranks is critical for population deployment. The authors show that a square-root scaling ($\alpha \propto \sqrt{r}$) law for the LoRA scaling parameter leads to transferable learning-rate regimes, minimizing manual re-tuning.

Beyond static LoRA, the authors advance the concept of stateful, dynamically writable adapters. $\delta$-mem, for example, implements an online recurrent associative memory in the adapter that tracks key–value associations, updating with a delta rule and injecting history-conditioned low-rank corrections into the frozen backbone during forward computation.

Figure 6: $\delta$-mem architecture couples an online memory state to low-rank corrections in the backbone attention computation, enabling efficient long-term adaptation.

Scale Out: Populations, Memory Laws, Simulation, and Diversity

Scaling out from individuals to millions of adapted models raises design questions of diversity, capacity, and system-level performance. Adapters as persistent local state provide mechanisms for preference, skill, and behavior continuity—critical for simulating real users, supporting memory, and enabling robust collective intelligence.

The empirical law for LoRA-adapter capacity, as measured via DishNameBenchmark, is clear: usable memory capacity lies between $r=16$0 and $r=16$1 tokens per trainable parameter, after which retrieval accuracy and behavioral integrity degrade predictably. MLP LoRA provides maximal parameter efficiency for memory under fixed model budgets.

Figure 7: DishNameBenchmark reveals a hard bound on LoRA memory: usable capacity is sharply limited relative to parameter count, with MLP adapters the most parameter-efficient.

The adaptive state is best deployed for durable behavioral policies, not rote fact storage; e.g., skill internalization leads to large improvements on procedural/agentic tasks like ALFWorld. A Qwen3-235B backbone plus a rank-32 LoRA adapter trained on skill-centric recipes improves validation score from 0.646 to 0.845 across categories.

Figure 8: On ALFWorld, large backbone plus skill-adapted LoRA raises average task score from $r=16$2 to $r=16$3.

Beyond individuals, per-user adapters circumvent the collapse of diversity seen in prompt-based user simulation. In OASIS, per-user LoRA adapters yield monotonic increases in action diversity, community count, and co-engagement modularity as population size grows. They support persistent behavioral heterogeneity unattainable via shared-policy models.

Figure 9: Scale Out: PEFT enables persistent local variation (individuality) and robust population-level utility within a shared base model.

Population-level diversity becomes a source of collective performance. A controlled model-count experiment shows that majority voting across $r=16$4 distinct LoRA adaptations improves accuracy nearly linearly in $r=16$5, reaching $r=16$6 at $r=16$7 (vs. a single-model baseline of $r=16$8). Gains produced by repeated sampling from the same model saturate, confirming that genuine policy diversity contributes value, not mere stochasticity.

Figure 10: Model-count scaling in majority voting—aggregate accuracy rises approximately linearly in $r=16$9 as more distinct adapters vote, outperforming repeated sampling.

Practical Infrastructure for PEFT Populations

Population-scale personalized models demand system-level support encompassing adapter identity, revision management, and resource residency control. The paper proposes MinT, a managed infrastructure stack that separates dense/MoE base residency from adapter mobility. MinT indexes adapter policies, orchestrates policy sessions and revisions, and supports fast, low-friction serving of bounded working sets (e.g., up to $32$0 cataloged adapters with sub-second cold-paging to active GPU batches of up to 64).

This infrastructure is essential for operationalizing the three axes: supporting intensive RL with prior provenance (Scale Up), adapter-only mobility (Scale Down), and residency/addressability separation (Scale Out).

Conclusion

The work provides a theoretically grounded and empirically validated multi-axis scaling theory of PEFT, arguing that individual and population-level persistent adaptation is only feasible through their careful integration. The principal limitations are the scale of empirical deployment and open questions at each axis: RL-specific PEFT theory, low-rank reliability, dynamic memory architectures, context-based adaptation signal extraction, and robustness of large populations under continual drift. The dominant implication is a shift from monolithic, universal assistants to persistent populations of specialized models—each efficiently individuated via small, composable, and persistent PEFT state atop strong foundation priors.

---

Reference: "On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters" (2606.02437)