Multiple-in-One Training Paradigm

• Multiple-in-One Training Paradigm is a training principle that structures one optimization run to yield multiple coupled models through shared parameterization.
• It employs methods such as shared-subnetwork co-evolution and weight-space parameterization to optimize compute resources and improve performance.
• Its applications span compression, multitask learning, and multimodal restoration, delivering benefits like regularization, efficient compute amortization, and versatile deployment.

Multiple-in-One Training Paradigm denotes a family of training designs in which one optimization process is structured to yield multiple coupled models, task tradeoffs, capacities, behaviors, or experimental outcomes, rather than a single isolated predictor. In the literature, the term covers shared-parameter teacher–student compression, one-stop co-training of high- and low-capacity models, continuous Pareto-subspace learning, multimodal and cross-domain unified models, online RL that turns one expensive state into multiple action-learning samples, and pretraining runs that host many controlled interventions simultaneously. The common denominator is not a single architecture, but a change in the unit of reuse: one parameterization, one training stream, one environment state, or one pretraining run is made to serve several objectives at once (Nath et al., 2020, Jiang et al., 2023, Dimitriadis et al., 2022, Liu et al., 2023, Gan et al., 8 Apr 2026, Bordt et al., 27 Sep 2025).

1. Genealogy and conceptual scope

Early formulations already separated the idea from ordinary single-task optimization. "One Model To Learn Them All" trained one shared model concurrently on ImageNet, multiple translation tasks, image captioning, speech recognition, and parsing, using small modality nets and a shared body with convolutional blocks, attention blocks, and sparsely-gated MoE layers (Kaiser et al., 2017). "Superposition of many models into one" then pushed the idea into parameter space by showing that multiple task-specific models can coexist in one parameter tensor through task-dependent contexts, so that one network behaves like many separate networks retrieved by unbinding (Cheung et al., 2019). In parallel, "One Big Net For Everything" framed the problem as lifelong consolidation: a single recurrent network, ONE, repeatedly learns new skills through RL, black-box optimization, artificial evolution, supervised learning, and unsupervised prediction, then compresses successful traces back into one shared substrate by replay and retraining (Schmidhuber, 2018).

Later work broadened the paradigm beyond standard DNN multitask learning. In statistical mechanics, "Parallel Learning by Multitasking Neural Networks" argued that multiple patterns can be learned at once because sparse patterns leave “unused” degrees of freedom, producing hierarchical and parallel regimes in a low-storage diluted Hopfield setting (Agliari et al., 2023). Subsequent neural-network papers instantiated the same high-level motif in compression, translation, graph foundation models, restoration, RL, and pretraining methodology. Taken together, these works show that “multiple-in-one” is best understood as a training principle: a single run is deliberately organized so that information acquired for one purpose is immediately available to others.

2. Recurrent mechanistic templates

A first template is shared-subnetwork co-evolution. In "Adjoined Networks," each convolution is replaced by an adjoined convolution in which the full network uses $W$ while the compressed network uses $W * M$, with the smaller model literally implemented as a masked structured subnetwork of the larger one. The joint objective

$$L(y,p,q) = -y\log p + \lambda(t)\, KL(p,q), \qquad \lambda(t)=\min\{4t^2,1\}$$

couples the two predictions so that the compressed model is trained “inside” the base network from the beginning, rather than as a post-hoc student (Nath et al., 2020).

A second template is weight-space parameterization of a solution family. In "Pareto Manifold Learning," the learned object is not one compromise MTL solution but a convex subspace

$$\mathcal{S}=\left\{\sum_{t=1}^{T}\alpha_t\boldsymbol{\theta}_t:\sum_{t=1}^T\alpha_t=1,\ \alpha_t\ge 0\right\},$$

with interpolated parameters $$\boldsymbol{\theta}=\boldsymbol{\alpha}^\top\boldsymbol{\Theta}$$ sampled from a Dirichlet distribution. One training run therefore yields a continuous Pareto-like family whose tradeoff can be modulated at inference (Dimitriadis et al., 2022).

A third template is contextual binding in parameter superposition. PSP stores

$W = \sum_{i=1}^{K} W_i C_i^{-1}$

and retrieves task $k$ by $\hat{W}_k = W C_k$, so that each task update is approximately projected back into its own context subspace. This makes the training dynamics themselves multiple-in-one: gradients superpose as well as forward passes, and interference is treated as near-zero-mean residual noise in high dimension (Cheung et al., 2019).

A fourth template is sample expansion around an expensive state. "Android Coach" replaces Single State Single Action with Single State Multiple Actions by sampling $k$ actions $\{a_t^1,\dots,a_t^k\}\sim\pi_\theta(\cdot\mid s_t)$ for one collected online state, evaluating them with a critic $W * M$0, and computing the Actor-Critic Leave-One-Out estimator

$W * M$1

Here the same emulator state is amortized over multiple policy-learning samples without extra environment interaction (Gan et al., 8 Apr 2026).

These constructions differ algebraically, but all redefine the atomic training object. The atomic object may be a mask-sharing layer, a convex hull in weight space, a task-conditioned superposition, or a state expanded into a group of action candidates; in each case, one training budget is made to produce more than one effective model update.

3. Representative realizations across domains

The paradigm now spans compression, capacity control, cross-domain unification, restoration, RL, and even methodological experimentation.

Domain	Reused training unit	Representative instantiation
Compression and capacity control	One parameterization for large and small models	AN, DAN, TSJT
Multitask and cross-domain modeling	One body or prompted format for many tasks	MultiModel, OFA
Restoration and multimodal pretraining	One backbone guided by multiple priors or alignments	Review Learning, LMDIR, ITO
Control and experimentation	One state or one pretraining run for many updates/outcomes	Android Coach, CMC, OLMo-2-1B-Exp

In graph learning, "One for All" standardizes node, link, and graph classification by converting heterogeneous data into text-attributed graphs, introducing nodes-of-interest, and appending prompt substructures containing a NOI prompt node and class nodes with textual descriptions. The result is not merely one encoder with multiple heads, but a single prompted graph representation supporting supervised, few-shot, and zero-shot inference across citation networks, molecular graphs, knowledge graphs, and wiki graphs (Liu et al., 2023).

In image restoration, two distinct multiple-in-one strategies appear. "Review Learning" trains a single restoration model sequentially across degradation datasets ordered by entropy difference, while replaying hard samples selected by high loss or top 20% entropy differences; the training schedule, rather than an inference-time prompt module, is the primary unifying mechanism (Su et al., 2024). "Training-Free Large Model Priors for Multiple-in-One Image Restoration" instead injects three priors—global degradation knowledge from GPT-4o, scene-aware contextual descriptions from GPT-4o encoded by CLIP, and fine-grained synthesized references from SDXL—through a query-based prompt encoder, degradation-aware transformer block, content-aware transformer block, and reference-based transformer block, enabling one restoration network to handle denoising, deraining, and low-light enhancement in a single-stage training regime (He et al., 2024).

In multimodal representation learning, "ITO: Images and Texts as One" combines multimodal multiple alignment with a lightweight training-time fusion module. Multiple image-text views provide denser supervision, while the fusion branch acts as a structural regularizer during training and is discarded at inference, preserving dual-encoder efficiency (Liu et al., 3 Mar 2026). In LLM post-training, "Cross-model Control" trains a tiny delta model alongside a frozen template LLM so that one learned logit-shift module can be ported to multiple user models with different parameter scales and vocabularies using PM-MinED token mapping (Wu et al., 2024).

These examples indicate that the paradigm is orthogonal to modality. The same design logic appears in CNN compression, MoE–dense co-training, graph prompting, restoration with large-model priors, image–text contrastive pretraining, and cross-model LLM control.

4. Empirical behavior and claimed advantages

In network compression, AN and DAN were presented as both compression and regularization methods. On ImageNet with ResNet-50 as the base network, AN-50-Small $W * M$2 reaches 75.1% top-1 accuracy with roughly 4× compression and about 2× FLOP reduction, AN-50-Small $W * M$3 reaches 71.82% with 2.2M parameters and 1.6 GFLOPs, DAN-50 reaches 73.33% with 3.49M parameters and 1.7 GFLOPs, and DAN-100 reaches 75.43% with 6.58M parameters and 2.15 GFLOPs. The regularization effect is also explicit: AN-Full ResNet-50 reaches 76.87% compared to 76.1% for standard training (Nath et al., 2020).

In multi-capacity co-training and AMTO-style optimization, the reported gains are similarly framed as benefits of coupled training trajectories. On WMT10, TSJT-shared MoE reaches 28.72 average BLEU in X→En versus 27.06 for single MoE, while TSJT-indep device reaches 20.04 versus 19.62 for the single device model; the paper attributes this to stronger early mutual guidance followed by separation once the submodels enter a “rather optimal region” (Jiang et al., 2023). In AMTO, the final model is selected by harmonic accuracy across multiple validation splits, and the framework outperforms STO in all tested cases; for example, OxfordPets with SqueezeNet improves from 82.90 to 84.34, and DenseNet-121 improves from 93.00 to 93.19 (Zhang et al., 2020).

Unified models also report broad empirical viability. OFA supports 0-shot performance across node-, link-, and graph-level settings, including 46.19 on ogbn-arxiv 5-way 0-shot, 70.20 on FB15K237 20-way 0-shot, and 35.67 on HIV 0-shot, using the same graph model across domains (Liu et al., 2023). Android Coach improves success rate from 31.9% to 39.4% on AndroidLab and from 32.8% to 41.1% on AndroidWorld, while attaining 1.4× higher training efficiency than SSSA methods PPO and GRPO at matched success rates (Gan et al., 8 Apr 2026). ITO improves average zero-shot accuracy by 2.6% over CLIP on Laion100M, and on DataComp-1B a ViT-B/16 linear probe reaches 89.18% average versus 87.62% for CLIP (Liu et al., 3 Mar 2026). In "Train Once, Answer All," the simultaneous interventions modify 3.7B tokens, only 1.8% of a 210B-token run, yet the final validation loss remains 2.6100 versus 2.6088 for the control and held-out benchmark accuracy remains 55.15% versus 55.51%, supporting the claim that many experiments can coexist in one run with minimal disturbance (Bordt et al., 27 Sep 2025).

Across these studies, the claimed advantages cluster around compute amortization, regularization, improved early optimization, retention of multiple capabilities, and the production of several deployable operating points from one run.

5. Distinctions, misconceptions, and limitations

A recurrent misconception is that multiple-in-one training is simply another name for distillation or conventional multitask learning. The literature is more specific. In AN, the student is not a post-hoc artifact but a masked subnetwork trained jointly with the teacher from the beginning, and the teacher is not frozen (Nath et al., 2020). In TSJT, the coupling is explicitly parallel and bidirectional, not serial and unidirectional as in KD; moreover, the paper argues that constant joint training can become harmful later in optimization, which is why the KL term is removed after the separation threshold is reached (Jiang et al., 2023).

Another misconception is that any shared model automatically qualifies. PaMaL differs from ordinary shared-bottom MTL because it aims to learn a continuous Pareto subspace rather than one predefined compromise point (Dimitriadis et al., 2022). OFA differs from ordinary multitask graph learning because it standardizes tasks through nodes-of-interest and prompt graphs instead of merely attaching different heads (Liu et al., 2023). CMC differs from model-specific PEFT because the learned object is a portable tiny delta model that alters logits across user LLMs rather than adapting each base model in its own parameter space (Wu et al., 2024).

The paradigm also has clear limits. OFA explicitly states that it currently does not handle regression well, and also notes that graph foundation data remain scarce compared with LLM corpora (Liu et al., 2023). Review Learning reports that excessive review can reintroduce the very “information pollution” it tries to avoid: 20% review is best in the reported ablation, while too much review can produce a mottled or speckled background (Su et al., 2024). CMC states that if the user model contains tokens from languages not covered by the delta model’s vocabulary, those tokens will not be adjusted (Wu et al., 2024). "Train Once, Answer All" identifies interaction between experiments as a potential confounder and further notes that CPDT does not work well for verbatim memorization, because that phenomenon depends on the absence of specific texts throughout training rather than a localized continual-pretraining perturbation (Bordt et al., 27 Sep 2025).

These caveats show that the paradigm is not a universal replacement for task-specific training. Its benefits depend on compatibility between the coupled objectives, the strength of the shared inductive structure, and the ability to diagnose negative transfer or interaction effects.

6. Theoretical interpretations and research significance

Several papers supply theoretical lenses for why multiple-in-one training may work. "Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks" shows that multitask pretraining induces a pseudo-contrastive objective over pairs of inputs, with

$W * M$4

acting as the cross-task agreement signal. Under the task-diversity assumption, the resulting update recovers the latent $W * M$5-dimensional signal subspace, and downstream generalization depends on $W * M$6 rather than the ambient dimension $W * M$7 (Collins et al., 2023). This provides a formal account of why one shared representation trained on many tasks can be more informative than one learned from a single task.

In the statistical-mechanics treatment of multitasking Hebbian networks, sparse patterns create room for simultaneous retrieval and learning of several patterns. The theory identifies hierarchical and parallel regimes, gives the ergodicity-breaking threshold

$W * M$8

and concludes that in the low-storage regime supervised and unsupervised training have the same asymptotic thresholds and macroscopic behavior (Agliari et al., 2023). This is a non-deep-learning instance of the same principle: sparsity or unused capacity can be converted into parallel learning rather than competition.

At the level of experimental methodology, "Train Once, Answer All" formalizes when one run can validly host many interventions by requiring, for experiment $W * M$9,

$$L(y,p,q) = -y\log p + \lambda(t)\, KL(p,q), \qquad \lambda(t)=\min\{4t^2,1\}$$0

so that the outcome of experiment $$L(y,p,q) = -y\log p + \lambda(t)\, KL(p,q), \qquad \lambda(t)=\min\{4t^2,1\}$$1 should be invariant to the presence of others (Bordt et al., 27 Sep 2025). This extends the paradigm beyond model training into research design itself.

This suggests a unifying interpretation. Multiple-in-one training is effective when the training process can expose shared algorithmic information without forcing incompatible objectives into destructive interference. The shared object may be filters, latent subspaces, prompt structures, replay buffers, critic-evaluated action groups, or experimental pockets in a pretraining corpus. A plausible implication is that the paradigm will continue to expand wherever compute, environment interaction, or annotation budgets make isolated training runs prohibitively expensive, but its methodological success will continue to hinge on diagnosing coupling structure rather than assuming it.