Unified Learning: A Common Substrate Approach

• Unified Learning is a design principle that unifies representations, objectives, architectures, and training dynamics across diverse tasks and domains.
• It leverages shared latent states, token spaces, and distillation techniques to enhance transfer efficiency, sample usage, and robustness in vision, reinforcement, and multimodal systems.
• Key challenges include optimization conflicts, coverage mismatches, and scalability concerns, which guide future directions for modular and cost-effective unified systems.

Unified learning denotes a family of research programs that seek a single organizing mechanism across heterogeneous settings: a shared encoder across tasks and domains, a unified latent state space across environments, a common token space across modalities, or one training procedure that supports sequential learning and forgetting. In vision, this appears as a “single deep neural network” for multiple tasks and domains (Li et al., 2022); in reinforcement learning, as a “latent unified state representation” shared across visually different domains (Hearn et al., 2022); in multimodal modeling, as a frozen modality-shared encoder or a single autoregressive token interface across text and images (Zhang et al., 2023, Tang et al., 27 Mar 2025); and in continual systems, as one framework that jointly handles continual learning and machine unlearning (Chatterjee et al., 2024, Huang et al., 21 May 2025). The term therefore does not identify one canonical formalism. It identifies a recurring objective: replacing fragmented, task-specific training pipelines with a common representational, architectural, or optimization substrate.

1. Conceptual scope and recurring design patterns

Across the literature, unified learning is instantiated through a small number of recurring abstractions. Some works unify representations, some unify objectives, some unify architectures, and some unify training dynamics. “Universal Representations” frames the problem as learning one shared encoder for either multiple tasks in one domain or one task across multiple domains (Li et al., 2022). USRA frames it as learning one domain-general latent state space across visually distinct but dynamically identical environments (Hearn et al., 2022). Meta-Transformer frames it as mapping many raw modalities into one shared token space processed by a frozen encoder (Zhang et al., 2023). UGen frames it as one decoder-only Transformer operating over one mixed vocabulary of text and image tokens (Tang et al., 27 Mar 2025). RIGL frames it as one holistic knowledge tracing problem spanning both independent and group learning (Yu et al., 2024).

Formulation	Shared object	Representative papers
Multi-task / multi-domain learning	Shared encoder or universal representation	(Li et al., 2022)
Domain-general RL	Shared latent state space	(Hearn et al., 2022)
Multimodal learning	Shared token or embedding space	(Zhang et al., 2023, Tang et al., 27 Mar 2025, Astruc et al., 13 Apr 2026)
Continual learning / unlearning	Shared update rule or distillation framework	(Chatterjee et al., 2024, Huang et al., 21 May 2025)
Structured social or educational modeling	Shared reciprocal state space across levels	(Yu et al., 2024)

This pattern suggests that unified learning is best understood as a structural principle rather than a single algorithmic family. The common move is to identify a level of abstraction at which heterogeneous observations, tasks, or update requests can be made commensurate.

2. Shared spaces: latent states, tokens, and embeddings

A dominant route to unified learning is the construction of a shared representational space. In USRA, raw observations $s^o \in \mathbb{S}^o$ are mapped to a latent space $\mathbb{S}^z$, then decomposed as

$$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$

where $\widehat{\mathbb{S}^z}$ is domain-specific and $\overline{\mathbb{S}^z}$ is domain-general (Hearn et al., 2022). The unified state representation is precisely the claim that semantically equivalent states across domains should share $\overline{s^z}$, allowing one policy to act on a single latent MDP despite visual variation. In the DeepMind Control Generalization Benchmark for Walker, this formulation yielded higher sample efficiency and 14.3% better domain adaptation performance than the best baseline (Hearn et al., 2022).

In multimodal learning, the shared space is often tokenized rather than latent-dynamical. Meta-Transformer maps 12 modalities into $D$-dimensional token sequences processed by one frozen ViT-style encoder, with the initial sequence

$$\boldsymbol{z}_0 = [ \boldsymbol{x}_{CLS}; \, \boldsymbol{E}_{\boldsymbol{x}_1}; \cdots; \boldsymbol{E}_{\boldsymbol{x}_n} ] + \boldsymbol{E}_{pos},$$

and standard Transformer blocks thereafter (Zhang et al., 2023). Its claim is stronger than ordinary multimodal transfer: the same frozen encoder is reused for text, image, point cloud, audio, video, X-Ray, infrared, hyperspectral, IMU, graph, tabular, and time-series data, without paired multimodal training data (Zhang et al., 2023). UGen makes a different but related move: both texts and images are represented as discrete token sequences, delimited by [SOS], [EOS], [SOI], and [EOI], so that understanding and generation reduce to next-token prediction in one vocabulary (Tang et al., 27 Mar 2025).

UNIGEOCLIP generalizes the shared-space idea to five geospatial modalities—SV, sat, DSM, txt, GPS—using modality-specific encoders $\phi^m$ and an all-to-all contrastive objective over ordered modality pairs: $$\mathcal{L} = \frac{1}{M^2} \sum_{(m,n) \in \mathcal{M}^2} \mathcal{L}_{m \mapsto n}.$$ Rather than choosing a pivot modality, it directly aligns every modality with every other modality in one unified embedding space (Astruc et al., 13 Apr 2026). This enables arbitrary cross-modal retrieval and yields a GPS encoder with mean $\mathbb{S}^z$0 of 57.0 on 27 socio-economic and environmental regressions, compared with 49.8 for GeoCLIP and 30.1 for SatCLIP in the reported setup (Astruc et al., 13 Apr 2026).

A more architectural version of shared space appears in GNN-based Unified Deep Learning, where each heterogeneous model is itself encoded as a graph and all such model-graphs are combined into a unified graph processed by a uGNN (Pala et al., 14 Aug 2025). Here the shared space is not an embedding of data instances but a graph learning space over parameterized model structures.

3. Unified objectives and optimization principles

A second route to unified learning is to preserve heterogeneity at the data level while enforcing a common optimization principle. “Universal Representations” does this by distilling multiple task- or domain-specific experts into one universal encoder with small adapters. Its training objective combines task loss, feature alignment, and optional prediction alignment: $\mathbb{S}^z$1 The unifying claim is that distillation provides a homogeneous alignment signal that mitigates multi-loss imbalance in naive joint training (Li et al., 2022). On Visual Decathlon, the single-encoder version reached 79.25% average accuracy and Decathlon score $\mathbb{S}^z$2 at 1× parameters, while the shared-backbone-plus-adapters version reached 80.52% and $\mathbb{S}^z$3 at 2× parameters (Li et al., 2022).

UniCL proposes a still more direct unification: one bidirectional contrastive loss over an image-text-label space. The image-to-text and text-to-image losses,

$\mathbb{S}^z$4

$\mathbb{S}^z$5

reduce to CLIP for unique image-caption labels and recover supervised cross-entropy as a special case under an embedding-matrix text encoder (Yang et al., 2022). This is a particularly explicit statement of unified learning: one objective subsumes supervised learning, supervised contrastive learning, and language-image contrastive learning (Yang et al., 2022).

At the most abstract end, “A unified theory of learning” defines learning as information compression and proposes the loss

$\mathbb{S}^z$6

where $\mathbb{S}^z$7 is a discrete memory and $\mathbb{S}^z$8 is the input event (Katayose, 2022). The paper interprets this as a free-energy-like functional whose expected value upper-bounds the expected self-information of the input distribution. This does not provide the empirical breadth of the other frameworks, but it articulates the strongest version of unified learning: one task-agnostic loss principle for any kind of input data (Katayose, 2022).

Transfer learning yields a complementary optimization view. “Adaptive Sample Aggregation in Transfer Learning” introduces the weak modulus of transfer

$\mathbb{S}^z$9

and the strong modulus

$$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$0

then proves adaptive procedures whose guarantees are stated directly in terms of these moduli rather than a specific divergence (Hanneke et al., 2024). This suggests a theoretical notion of unified learning in which discrepancies, Wasserstein bounds, covariance ratios, and transfer exponents are viewed as special upper bounds on a common transfer object.

4. Continual, reciprocal, and structurally coupled unification

Unified learning also appears in settings where the challenge is not multimodality but the coexistence of multiple structural levels or temporal update types. In RIGL, the core problem is “holistic knowledge tracing” over independent student learning and group learning. The model builds aligned time-frame embeddings for students and groups, performs reciprocal enhancement in both directions, models their dynamic relations with a graph, and stabilizes training with a bias-aware contrastive loss (Yu et al., 2024). The group-to-individual and individual-to-group updates are explicit: $$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$1

$$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$2

Across four real-world educational datasets, RIGL reports about +4.01% on individual AUC/ACC and about +20.32% improvement on group RMSE/MAE over the best baseline (Yu et al., 2024). The unified aspect is not merely multitasking; it is the claim that individual and group processes are mutually constitutive states of one model.

Unified-QG applies the same principle to task formats and time. It converts answer-extraction, answer-abstraction, multi-choice, and boolean QG into one text-to-text format and uses STRIDER—Similarity Regularized Difficult Example Replay—to support continual lifelong learning across eight QG datasets (Yuan et al., 2022). The continual metrics are explicit: Unified-QG reports $$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$3 BLEU-4 of 15.69 versus 15.67 for Multitask-QG, and $$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$4 BLEU-4 of 22.54 versus 20.36 for Multitask-QG (Yuan et al., 2022). A single trained Unified-QG model also improves eight QA systems when used to generate synthetic QA data (Yuan et al., 2022).

The continual learning–unlearning literature pushes unification further by treating learning and forgetting as coupled updates. UniCLUN uses controlled knowledge distillation with a student, a CL teacher, and a bad teacher, switching between CL and UL losses depending on task type (Chatterjee et al., 2024). UG-CLU gives the most explicit optimization synthesis, decomposing approximate CLU updates into four components: learning new knowledge, unlearning targeted data, preserving existing knowledge, and modulation via weight saliency (Huang et al., 21 May 2025). Its KL-based formulation and remain-preserved manifold lead to updates of the form

$$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$5

with a practical fast–slow approximation in the implemented algorithm (Huang et al., 21 May 2025). On task-aware CLU for CIFAR-10, UG-CLU reports LA 91.72%, UA $$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$6, MIA $$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$7, and the lowest KL among compared methods (Huang et al., 21 May 2025). This is unified learning in a procedural sense: one update rule for acquisition, deletion, and retention.

5. Embodied, reinforcement, and policy-level unified learning

In reinforcement learning and embodied control, unified learning usually denotes one policy or one state space spanning many environments or tasks. USRA learns one domain-general latent state from source-domain images augmented into virtual domains, combining a cycle-consistent VAE with SVEA-style Q-value consistency. Its pretraining objective is

$$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$8

followed by fine-tuning with $$\mathbb{S}^z = (\widehat{\mathbb{S}^z}, \overline{\mathbb{S}^z}),$$9 alone (Hearn et al., 2022). On DMControl-GB Walker, USRA reached train/eval returns of 949/949 on Color (Easy), 948 on Color (Hard), 862 on Video (Easy), and 245 on Video (Hard), compared with SVEA’s 892/888/871/703/202 and LUSR’s much lower scores (Hearn et al., 2022). The conceptual unification is that the policy and critic operate only on the shared domain-general latent.

PolyTask tackles unification at the policy level. It first learns task-specific experts with demonstration-guided RL, then distills them offline into one conditioned policy via Behavior Distillation: $\widehat{\mathbb{S}^z}$0 This avoids joint multi-task RL during interactive learning and makes lifelong skill accumulation possible without concurrent access to old environments (Haldar et al., 2023). On Meta-World, PolyTask reports 14.6 effective tasks versus 12.0 for MTRL-Demo; on FrankaKitchen, 4.5 versus 2.6; and on a six-task real-robot suite, the multi-task policy reaches 5.2/6 effective tasks (Haldar et al., 2023). Here the unification mechanism is distillation rather than a shared latent state.

UniRL-Zero extends the idea to mixed discrete–continuous multimodal policies. It defines a unified model containing a frozen multimodal LM expert and a diffusion model expert, then applies GRPO-style RL over joint LM token trajectories and DM denoising trajectories (Wang et al., 20 Oct 2025). The six scenarios—text reasoning, multimodal reasoning, text-to-image, image editing, CoT-enhanced text-to-image, and reflective image generation—are treated as a structured space of unified tasks. On GenEval, the base model scores 0.69 overall, T2I-RL scores 0.80, and CoT-enhanced T2I-RL scores 0.85, with especially strong gains in counting and color attribution (Wang et al., 20 Oct 2025). This suggests a more general sense of unified learning in which reasoning and generation are not merely colocated in one architecture but optimized as one decision process.

6. Scaling strategies, limitations, and open directions

The empirical record shows that unified learning can improve transfer, sample efficiency, and robustness, but it also reveals recurring constraints. One recurrent issue is optimization conflict. “Universal Representations” motivates distillation precisely because naive joint optimization of heterogeneous task losses often yields unbalanced training and poor average performance (Li et al., 2022). UGen diagnoses a related failure mode in unified autoregressive multimodal training: as the visual vocabulary grows, perplexity rises sharply, and a vanilla unified AR model underperforms task-specific AR models by 8.1–23.9% across tasks; progressive vocabulary learning recovers much of this gap and yields a 13.3% overall improvement over the vanilla unified AR baseline (Tang et al., 27 Mar 2025). STAR addresses an analogous problem by freezing a strong understanding backbone and progressively stacking isomorphic AR modules for generation and editing, reporting state-of-the-art 0.91 on GenEval, 87.44 on DPG-Bench, and 4.34 on ImgEdit (Qin et al., 15 Dec 2025).

A second recurring issue is coverage mismatch between the unifying mechanism and the real deployment shift. USRA assumes shifts are mostly visual and that reward and dynamics are unchanged (Hearn et al., 2022). UNIGEOCLIP notes limitations in geographic coverage, temporal scope, and modality availability, even while showing gains from all-to-all alignment across sat, SV, DSM, text, and GPS (Astruc et al., 13 Apr 2026). Meta-Transformer demonstrates broad reuse of a frozen image-pretrained encoder, but its graph results on PCQM4M-LSC are much weaker than specialized graph models, indicating that generic tokenization alone does not erase all structural mismatches (Zhang et al., 2023).

A third issue is cost and memory. Distillation-based universal representations require teacher networks during training (Li et al., 2022). PolyTask requires storing replay buffers for all tasks (Haldar et al., 2023). UniRL-Zero notes reward bias, scale constraints, and the expense of RL on diffusion models (Wang et al., 20 Oct 2025). GNN-based unified deep learning raises a different scalability concern: every model is converted into a graph, and the unified graph can become large as the number of models and domains grows (Pala et al., 14 Aug 2025).

These limitations indicate that unified learning is not reducible to the slogan “one model for everything.” The more durable lesson is narrower and more technical. Successful unified systems typically introduce a carefully chosen common substrate—distillation targets, token space, latent state, graph space, or KL geometry—while preserving enough modularity to prevent destructive interference. The open directions reported across the literature are correspondingly consistent: multi-task over multi-domain vision (Li et al., 2022), multimodal unified representation for RL (Hearn et al., 2022), more modalities and richer reward models for unified RL (Wang et al., 20 Oct 2025), global and temporal extensions for geospatial alignment (Astruc et al., 13 Apr 2026), and broader multimodal expansion of unified AR models (Tang et al., 27 Mar 2025). Taken together, these works suggest that unified learning is becoming a general design language for systems that must share structure without collapsing heterogeneity.