---
title: Full Unified Perception Paradigm
url: https://www.emergentmind.com/topics/full-unified-perception
type: topic
---

# Full Unified Perception Paradigm

Full Unified Perception is a research paradigm that treats scene understanding as a single, shared inference problem rather than a chain of isolated modules. In its most explicit formalization, the autonomous-vehicle survey defines **Full-Unified Perception (FUP)** as the paradigm that “primarily integrates scene understanding (detection, tracking, prediction) within a single module,” while also allowing incorporation of localization, and frames it as the highest level of task integration that still retains interpretable perception outputs [2508.20892]. Across adjacent literatures, the same expression is used more broadly for frameworks that jointly model appearance and geometry, multiple sensing modalities, or perception and action within one representation-learning system, so the term denotes a family of tightly related unification programs rather than a single architecture class [2503.13587] [2606.26196].

## 1. Definition and taxonomic position

In the survey taxonomy for autonomous vehicle software, unified perception extends modular perception by integrating scene-understanding tasks within a shared architecture while retaining interpretable outputs [2508.20892]. The taxonomy separates three paradigms according to the scope of integration: Early Unified Perception unifies detection and tracking, Late Unified Perception unifies tracking and prediction, and Full Unified Perception unifies detection, tracking, and prediction. The distinctive property of FUP is not merely that multiple tasks share a backbone, but that they are solved as a holistic scene-understanding problem from raw sensor input.

| Paradigm | Integrated tasks | Characteristic input |
|---|---|---|
| EUP | detection + tracking | raw sensor input |
| LUP | tracking + prediction | intermediate detection results |
| FUP | detection + tracking + prediction | raw sensor input |

This definition is narrower than many informal uses of “unified perception.” In the survey, FUP is specifically about the fusion of the three canonical scene-understanding tasks inside one perception module [2508.20892]. By contrast, several recent papers use “full unified perception” to describe broader couplings such as future generation plus future depth in driving world models, equal-modality tracking across RGB-X settings, or unified vision-language perception in multimodal large language models [2503.13587] [2502.06583] [2606.26196]. A plausible implication is that the phrase has evolved from a driving-specific taxonomic label into a more general description of architectures that collapse formerly separate perceptual stages into one jointly optimized system.

## 2. Representation flow and architectural organization

A central issue in FUP is representation flow. The survey emphasizes that the representation type “plays a crucial role in information sharing across sub-tasks, directly affecting explainability, interconnectivity … and likely system performance,” and notes that FUP systems may operate on raw sensor data while using intermediate representations such as bounding boxes, occupancy grids, or latent features, and may emit trajectories, occupancy, or flow [2508.20892]. This is why FUP methods differ not only by task set, but by whether they are object-centric, occupancy-centric, or latent-state-centric.

In vision-centric 3D perception, UniVision provides a concrete unification template for two major tasks that are usually separated: occupancy prediction and 3D object detection. Its architecture uses an explicit-implicit view transform, a voxel branch for local 3D reasoning, a BEV branch for global planar reasoning, and cross-representation interaction so that voxel-level and BEV-level features enrich each other [2401.06994]. The method’s “join-divide-join” pattern is representative of unified perception systems that begin with shared image features, branch into task-appropriate representations, and then re-couple them before prediction. This suggests that full unification often requires structured heterogeneity rather than a single homogeneous feature space.

Other work makes the same point from an optimization perspective. FULLER argues that naïvely extending multimodal fusion to multi-task learning is ineffective because of task conflict and modality bias, and therefore introduces multi-level gradient calibration: inter-gradient calibration at the shared backbone’s last layer and intra-gradient calibration before gradients enter modality-specific branches [2307.16617]. The paper’s gains—an absolute **14.4% mIoU** improvement on map segmentation and **1.4% mAP** improvement on 3D detection on nuScenes—indicate that shared representation alone is insufficient; unified perception also depends on controlling how tasks and modalities compete during optimization [2307.16617].

A more recent driving VLA formulation, UniDriveVLA, makes the same architectural point in a different form. It argues that a shared parameter space induces a perception–reasoning dilemma, and addresses this with a Mixture-of-Transformers composed of three experts for driving understanding, scene perception, and action planning, coordinated through masked joint attention [2604.02190]. Here, unification is achieved not by forcing semantic reasoning and spatial perception into one decoder, but by expert decoupling plus controlled interaction. This suggests that “full” unification need not imply parameter uniformity.

## 3. Tracking, prediction, and the explicit–implicit distinction

Within the FUP taxonomy, the treatment of tracking is fundamental. The survey distinguishes **explicit tracking**, which assigns persistent identities through hard one-to-one associations, from **implicit tracking**, which avoids persistent identity assignment and instead maintains temporal continuity in latent representations [2508.20892]. This distinction matters because FUP unifies tracking with prediction. Once prediction is integrated, post-network association is generally incompatible with joint reasoning: the survey states that in LUP and FUP, post hoc association “would break the joint reasoning process and prevent shared optimization between tasks” [2508.20892].

Trajectory-output FUP methods illustrate both possibilities. Methods such as ViP3D and ODTP maintain track-consistent trajectories with explicit tracking, while PIP, FaF, IntentNet, SpAGNN, ILVM, and DeTra represent more latent or implicit approaches in which temporally coherent scene state is carried in learned representations rather than explicit identity chains [2508.20892]. Occupancy-output FUP methods push this further: BEVerse, StretchBEV, TBP-Former, ImplicitO, Occ4Cast, and related systems predict future occupancy or scene evolution and can therefore replace object-wise identity tracking with dense spatiotemporal continuity [2508.20892].

This distinction reappears outside the survey’s driving taxonomy. UniFuture formulates a driving world model that jointly predicts future RGB and future depth within a shared latent space, using Dual-Latent Sharing and Multi-scale Latent Interaction, and at test time predicts future image-depth pairs from only the current image [2503.13587]. Its notion of “full unified perception” is not detection–tracking–prediction in the survey sense; rather, it is the joint forecasting of future appearance and future geometry with structural consistency. APTrack makes a similar move in unified multi-modal tracking: it rejects RGB-dominant fusion, uses equal modeling of RGB and the auxiliary modality with shared weights, and introduces an Adaptive Modality Interaction module that bridges modalities through learnable tokens [2502.06583]. In both cases, the literature treats temporal continuity and modality continuity as part of the same unification problem.

## 4. Extensions beyond autonomous-vehicle scene understanding

The broader literature generalizes full unified perception beyond tracking-centric scene understanding. In multimodal large language models, the survey on vision-language perception defines perception as an intrinsic, inseparable vision-language capability and identifies three requirements for a truly unified paradigm: a unified intermediate representation, a unified task interface, and unified resource-awareness [2606.26196]. This formulation shifts the emphasis from object pipelines to region- and instance-level grounding, structured outputs, tool use, and adaptive perception–reasoning loops. A plausible implication is that FUP, in this setting, is less about which tasks are unified than about whether representation, reasoning, and action are organized as one modality-coupled process.

VisionReasoner gives an operational version of that claim. It reformulates detection, segmentation, and counting as variants of multi-object cognition, uses one shared LVLM framework, generates a structured reasoning process before the final output, and trains with GRPO using format rewards and localization rewards [2505.12081]. The model is evaluated on ten benchmarks spanning detection, segmentation, and counting, and reports relative improvements over Qwen2.5VL of **29.1%** on COCO detection, **22.1%** on ReasonSeg, and **15.3%** on CountBench [2505.12081]. Here, full unified perception denotes a reasoning-driven perceptual model rather than a perception stack with separate heads.

Agentic medical analysis offers a different extension. 3DMedAgent is a tool-augmented 2D MLLM agent for 3D CT analysis that replaces one-shot 3D prediction with a query-adaptive evidence-seeking loop composed of Organ-Aware Memory Initialization, Coarse-to-Fine Lesion Targeting, and Think-with-1-Slice Loop, all mediated by long-term structured memory [2602.18064]. The unification lies in the transition from global volume reasoning to regional localization, then to slice-level verification and structured textual understanding. In this usage, perception is unified with memory, tool invocation, and multistep reasoning.

Robotics work extends the concept again. CUBic frames bimanual manipulation as a unified perceptual modeling problem, with unidirectional perception aggregation, bidirectional perception coordination through two codebooks with shared mapping, and a unified perception-to-control diffusion policy [2605.13452]. Its results on RoboTwin—**51.8% average success**, versus **39.8%** for DP3 and **38.5%** for DP—support the claim that shared tokenized latent structure can jointly support arm-specific independence and cross-arm coordination [2605.13452]. This suggests that, outside classical scene understanding, full unified perception increasingly serves as a bridge concept between perception and control.

## 5. Optimization, supervision, and evaluation practice

A recurring result across the literature is that naïve joint training is usually insufficient. NeurAll, an early automotive precursor, already argued for a shared encoder with task-specific decoders and showed that multi-stream learning and auxiliary learning can improve performance while maintaining nearly unchanged parameter count; in its two-stream three-task setting, the product/geometric-mean loss outperformed naïve summation on KITTI and Cityscapes [1902.03589]. Later work makes the same point more sharply. UniVision states that without its progressive loss weight adjustment strategy, the unified model cannot converge, and combines this schedule with joint occupancy-detection augmentation to stabilize training [2401.06994]. FULLER attributes failure to task conflict and modality bias and addresses both directly at the gradient level [2307.16617].

Training-time alignment has become another common strategy. ALN-P3 introduces Perception Alignment, Prediction Alignment, and Planning Alignment to align a fast BEV-based driving stack with a slow language-driven module during training only, with no additional inference costs [2505.15158]. Its reported gains span planning, captioning, and QA: on nuScenes planning it reaches **0.16% average** collision rate, and on TOD3Cap it reaches **341.9 CIDEr** [2505.15158]. This is a form of unified perception in which natural language acts as a semantic teacher for the full perception–prediction–planning stack.

Evaluation remains heterogeneous. In driving FUP, the survey notes a lack of clear unified-versus-modular benchmark comparison, limited closed-loop validation, and uncertainty over the best representation and tracking style [2508.20892]. Outside driving, benchmarks are even less commensurate: UniFuture reports FID, FVD, and depth metrics on nuScenes [2503.13587]; APTrack evaluates on LasHeR, RGBT234, VisEvent, DepthTrack, and VOT-RGBD2022 [2502.06583]; VisionReasoner uses AP, gIoU, and counting accuracy across ten benchmarks [2505.12081]; 3DMedAgent introduces DeepChestVQA with **1,020 VQA pairs** across **17 capability dimensions** [2602.18064]. This suggests that unified perception currently lacks a domain-independent evaluation protocol, even when the underlying design principle is similar.

## 6. Common misconceptions, limits, and open questions

A common misconception is that full unified perception is simply multi-task learning. The literature indicates a stricter requirement: joint optimization must preserve or improve inter-task synergy, temporal consistency, and explainability rather than merely attaching multiple heads to one backbone [2508.20892]. Another misconception is that unification requires explicit identity tracking. The survey explicitly allows both explicit and implicit tracking in FUP, and occupancy-based methods demonstrate that dense scene evolution can substitute for object-ID pipelines in some settings [2508.20892]. A third misconception is that unification implies one undifferentiated parameter space. UniDriveVLA and CUBic both argue the opposite: decoupled experts or codebooks may be necessary so that coordination emerges without representational collapse [2604.02190] [2605.13452].

The open problems are correspondingly structural. In autonomous driving, the survey highlights modality imbalance, uncertainty over whether implicit tracking can match explicit methods under long occlusions and identity ambiguity, limited open-source availability, and the difficulty of safety, explainability, and verification in integrated systems [2508.20892]. In multimodal large language models, the vision-language survey identifies dependence on high-quality curated data, the lack of general and fine-grained reward models, and the high computational cost of dynamic or tool-augmented perception, while also raising the deeper question of whether a truly unified representation space across modalities exists at all [2606.26196]. UniFuture adds a task-specific limitation: semantic understanding is not explicitly modeled, even though appearance and geometry are jointly predicted [2503.13587].

Taken together, these results indicate that Full Unified Perception is best understood as a spectrum of architectures that replace modular interfaces with shared, task-coupled representations while still preserving enough structure to support reasoning, supervision, and validation. In the narrow autonomous-vehicle sense, it denotes the single-module integration of detection, tracking, and prediction from raw sensor data [2508.20892]. In the broader contemporary literature, it increasingly denotes a more ambitious condition: perception, memory, modality fusion, geometry, and action are treated as different operational views of one coordinated system rather than as separate software stages [2503.13587] [2602.18064] [2604.02190].

Source: https://www.emergentmind.com/topics/full-unified-perception