Full Unified Perception Paradigm
- Full Unified Perception is a paradigm that integrates detection, tracking, and prediction into a single module using shared representations from raw sensor data.
- It employs structured heterogeneity with distinct components like object-centric and occupancy-centric views to handle multiple tasks effectively.
- Unified optimization strategies such as multi-level gradient calibration and alignment techniques enhance inter-task synergy and improve performance metrics.
Full Unified Perception is a research paradigm that treats scene understanding as a single, shared inference problem rather than a chain of isolated modules. In its most explicit formalization, the autonomous-vehicle survey defines Full-Unified Perception (FUP) as the paradigm that “primarily integrates scene understanding (detection, tracking, prediction) within a single module,” while also allowing incorporation of localization, and frames it as the highest level of task integration that still retains interpretable perception outputs (Stratil et al., 28 Aug 2025). Across adjacent literatures, the same expression is used more broadly for frameworks that jointly model appearance and geometry, multiple sensing modalities, or perception and action within one representation-learning system, so the term denotes a family of tightly related unification programs rather than a single architecture class (Liang et al., 17 Mar 2025, Sun et al., 24 Jun 2026).
1. Definition and taxonomic position
In the survey taxonomy for autonomous vehicle software, unified perception extends modular perception by integrating scene-understanding tasks within a shared architecture while retaining interpretable outputs (Stratil et al., 28 Aug 2025). The taxonomy separates three paradigms according to the scope of integration: Early Unified Perception unifies detection and tracking, Late Unified Perception unifies tracking and prediction, and Full Unified Perception unifies detection, tracking, and prediction. The distinctive property of FUP is not merely that multiple tasks share a backbone, but that they are solved as a holistic scene-understanding problem from raw sensor input.
| Paradigm | Integrated tasks | Characteristic input |
|---|---|---|
| EUP | detection + tracking | raw sensor input |
| LUP | tracking + prediction | intermediate detection results |
| FUP | detection + tracking + prediction | raw sensor input |
This definition is narrower than many informal uses of “unified perception.” In the survey, FUP is specifically about the fusion of the three canonical scene-understanding tasks inside one perception module (Stratil et al., 28 Aug 2025). By contrast, several papers use “full unified perception” to describe broader couplings such as future generation plus future depth in driving world models, equal-modality tracking across RGB-X settings, or unified vision-language perception in multimodal LLMs (Liang et al., 17 Mar 2025, Hu et al., 10 Feb 2025, Sun et al., 24 Jun 2026). A plausible implication is that the phrase has evolved from a driving-specific taxonomic label into a more general description of architectures that collapse formerly separate perceptual stages into one jointly optimized system.
2. Representation flow and architectural organization
A central issue in FUP is representation flow. The survey emphasizes that the representation type “plays a crucial role in information sharing across sub-tasks, directly affecting explainability, interconnectivity … and likely system performance,” and notes that FUP systems may operate on raw sensor data while using intermediate representations such as bounding boxes, occupancy grids, or latent features, and may emit trajectories, occupancy, or flow (Stratil et al., 28 Aug 2025). This is why FUP methods differ not only by task set, but by whether they are object-centric, occupancy-centric, or latent-state-centric.
In vision-centric 3D perception, UniVision provides a concrete unification template for two major tasks that are usually separated: occupancy prediction and 3D object detection. Its architecture uses an explicit-implicit view transform, a voxel branch for local 3D reasoning, a BEV branch for global planar reasoning, and cross-representation interaction so that voxel-level and BEV-level features enrich each other (Hong et al., 2024). The method’s “join-divide-join” pattern is representative of unified perception systems that begin with shared image features, branch into task-appropriate representations, and then re-couple them before prediction. This suggests that full unification often requires structured heterogeneity rather than a single homogeneous feature space.
Other work makes the same point from an optimization perspective. FULLER argues that naïvely extending multimodal fusion to multi-task learning is ineffective because of task conflict and modality bias, and therefore introduces multi-level gradient calibration: inter-gradient calibration at the shared backbone’s last layer and intra-gradient calibration before gradients enter modality-specific branches (Huang et al., 2023). The paper’s gains—an absolute 14.4% mIoU improvement on map segmentation and 1.4% mAP improvement on 3D detection on nuScenes—indicate that shared representation alone is insufficient; unified perception also depends on controlling how tasks and modalities compete during optimization (Huang et al., 2023).
A more recent driving VLA formulation, UniDriveVLA, makes the same architectural point in a different form. It argues that a shared parameter space induces a perception–reasoning dilemma, and addresses this with a Mixture-of-Transformers composed of three experts for driving understanding, scene perception, and action planning, coordinated through masked joint attention (Li et al., 2 Apr 2026). Here, unification is achieved not by forcing semantic reasoning and spatial perception into one decoder, but by expert decoupling plus controlled interaction. This suggests that “full” unification need not imply parameter uniformity.
3. Tracking, prediction, and the explicit–implicit distinction
Within the FUP taxonomy, the treatment of tracking is fundamental. The survey distinguishes explicit tracking, which assigns persistent identities through hard one-to-one associations, from implicit tracking, which avoids persistent identity assignment and instead maintains temporal continuity in latent representations (Stratil et al., 28 Aug 2025). This distinction matters because FUP unifies tracking with prediction. Once prediction is integrated, post-network association is generally incompatible with joint reasoning: the survey states that in LUP and FUP, post hoc association “would break the joint reasoning process and prevent shared optimization between tasks” (Stratil et al., 28 Aug 2025).
Trajectory-output FUP methods illustrate both possibilities. Methods such as ViP3D and ODTP maintain track-consistent trajectories with explicit tracking, while PIP, FaF, IntentNet, SpAGNN, ILVM, and DeTra represent more latent or implicit approaches in which temporally coherent scene state is carried in learned representations rather than explicit identity chains (Stratil et al., 28 Aug 2025). Occupancy-output FUP methods push this further: BEVerse, StretchBEV, TBP-Former, ImplicitO, Occ4Cast, and related systems predict future occupancy or scene evolution and can therefore replace object-wise identity tracking with dense spatiotemporal continuity (Stratil et al., 28 Aug 2025).
This distinction reappears outside the survey’s driving taxonomy. UniFuture formulates a driving world model that jointly predicts future RGB and future depth within a shared latent space, using Dual-Latent Sharing and Multi-scale Latent Interaction, and at test time predicts future image-depth pairs from only the current image (Liang et al., 17 Mar 2025). Its notion of “full unified perception” is not detection–tracking–prediction in the survey sense; rather, it is the joint forecasting of future appearance and future geometry with structural consistency. APTrack makes a similar move in unified multi-modal tracking: it rejects RGB-dominant fusion, uses equal modeling of RGB and the auxiliary modality with shared weights, and introduces an Adaptive Modality Interaction module that bridges modalities through learnable tokens (Hu et al., 10 Feb 2025). In both cases, the literature treats temporal continuity and modality continuity as part of the same unification problem.
4. Extensions beyond autonomous-vehicle scene understanding
The broader literature generalizes full unified perception beyond tracking-centric scene understanding. In multimodal LLMs, the survey on vision-language perception defines perception as an intrinsic, inseparable vision-language capability and identifies three requirements for a truly unified paradigm: a unified intermediate representation, a unified task interface, and unified resource-awareness (Sun et al., 24 Jun 2026). This formulation shifts the emphasis from object pipelines to region- and instance-level grounding, structured outputs, tool use, and adaptive perception–reasoning loops. A plausible implication is that FUP, in this setting, is less about which tasks are unified than about whether representation, reasoning, and action are organized as one modality-coupled process.
VisionReasoner gives an operational version of that claim. It reformulates detection, segmentation, and counting as variants of multi-object cognition, uses one shared LVLM framework, generates a structured reasoning process before the final output, and trains with GRPO using format rewards and localization rewards (Liu et al., 17 May 2025). The model is evaluated on ten benchmarks spanning detection, segmentation, and counting, and reports relative improvements over Qwen2.5VL of 29.1% on COCO detection, 22.1% on ReasonSeg, and 15.3% on CountBench (Liu et al., 17 May 2025). Here, full unified perception denotes a reasoning-driven perceptual model rather than a perception stack with separate heads.
Agentic medical analysis offers a different extension. 3DMedAgent is a tool-augmented 2D MLLM agent for 3D CT analysis that replaces one-shot 3D prediction with a query-adaptive evidence-seeking loop composed of Organ-Aware Memory Initialization, Coarse-to-Fine Lesion Targeting, and Think-with-1-Slice Loop, all mediated by long-term structured memory (Wang et al., 20 Feb 2026). The unification lies in the transition from global volume reasoning to regional localization, then to slice-level verification and structured textual understanding. In this usage, perception is unified with memory, tool invocation, and multistep reasoning.
Robotics work extends the concept again. CUBic frames bimanual manipulation as a unified perceptual modeling problem, with unidirectional perception aggregation, bidirectional perception coordination through two codebooks with shared mapping, and a unified perception-to-control diffusion policy (Wang et al., 13 May 2026). Its results on RoboTwin—51.8% average success, versus 39.8% for DP3 and 38.5% for DP—support the claim that shared tokenized latent structure can jointly support arm-specific independence and cross-arm coordination (Wang et al., 13 May 2026). This suggests that, outside classical scene understanding, full unified perception increasingly serves as a bridge concept between perception and control.
5. Optimization, supervision, and evaluation practice
A recurring result across the literature is that naïve joint training is usually insufficient. NeurAll, an early automotive precursor, already argued for a shared encoder with task-specific decoders and showed that multi-stream learning and auxiliary learning can improve performance while maintaining nearly unchanged parameter count; in its two-stream three-task setting, the product/geometric-mean loss outperformed naïve summation on KITTI and Cityscapes (Sistu et al., 2019). Later work makes the same point more sharply. UniVision states that without its progressive loss weight adjustment strategy, the unified model cannot converge, and combines this schedule with joint occupancy-detection augmentation to stabilize training (Hong et al., 2024). FULLER attributes failure to task conflict and modality bias and addresses both directly at the gradient level (Huang et al., 2023).
Training-time alignment has become another common strategy. ALN-P3 introduces Perception Alignment, Prediction Alignment, and Planning Alignment to align a fast BEV-based driving stack with a slow language-driven module during training only, with no additional inference costs (Ma et al., 21 May 2025). Its reported gains span planning, captioning, and QA: on nuScenes planning it reaches 0.16% average collision rate, and on TOD3Cap it reaches 341.9 CIDEr (Ma et al., 21 May 2025). This is a form of unified perception in which natural language acts as a semantic teacher for the full perception–prediction–planning stack.
Evaluation remains heterogeneous. In driving FUP, the survey notes a lack of clear unified-versus-modular benchmark comparison, limited closed-loop validation, and uncertainty over the best representation and tracking style (Stratil et al., 28 Aug 2025). Outside driving, benchmarks are even less commensurate: UniFuture reports FID, FVD, and depth metrics on nuScenes (Liang et al., 17 Mar 2025); APTrack evaluates on LasHeR, RGBT234, VisEvent, DepthTrack, and VOT-RGBD2022 (Hu et al., 10 Feb 2025); VisionReasoner uses AP, gIoU, and counting accuracy across ten benchmarks (Liu et al., 17 May 2025); 3DMedAgent introduces DeepChestVQA with 1,020 VQA pairs across 17 capability dimensions (Wang et al., 20 Feb 2026). This suggests that unified perception currently lacks a domain-independent evaluation protocol, even when the underlying design principle is similar.
6. Common misconceptions, limits, and open questions
A common misconception is that full unified perception is simply multi-task learning. The literature indicates a stricter requirement: joint optimization must preserve or improve inter-task synergy, temporal consistency, and explainability rather than merely attaching multiple heads to one backbone (Stratil et al., 28 Aug 2025). Another misconception is that unification requires explicit identity tracking. The survey explicitly allows both explicit and implicit tracking in FUP, and occupancy-based methods demonstrate that dense scene evolution can substitute for object-ID pipelines in some settings (Stratil et al., 28 Aug 2025). A third misconception is that unification implies one undifferentiated parameter space. UniDriveVLA and CUBic both argue the opposite: decoupled experts or codebooks may be necessary so that coordination emerges without representational collapse (Li et al., 2 Apr 2026, Wang et al., 13 May 2026).
The open problems are correspondingly structural. In autonomous driving, the survey highlights modality imbalance, uncertainty over whether implicit tracking can match explicit methods under long occlusions and identity ambiguity, limited open-source availability, and the difficulty of safety, explainability, and verification in integrated systems (Stratil et al., 28 Aug 2025). In multimodal LLMs, the vision-language survey identifies dependence on high-quality curated data, the lack of general and fine-grained reward models, and the high computational cost of dynamic or tool-augmented perception, while also raising the deeper question of whether a truly unified representation space across modalities exists at all (Sun et al., 24 Jun 2026). UniFuture adds a task-specific limitation: semantic understanding is not explicitly modeled, even though appearance and geometry are jointly predicted (Liang et al., 17 Mar 2025).
Taken together, these results indicate that Full Unified Perception is best understood as a spectrum of architectures that replace modular interfaces with shared, task-coupled representations while still preserving enough structure to support reasoning, supervision, and validation. In the narrow autonomous-vehicle sense, it denotes the single-module integration of detection, tracking, and prediction from raw sensor data (Stratil et al., 28 Aug 2025). In the broader contemporary literature, it increasingly denotes a more ambitious condition: perception, memory, modality fusion, geometry, and action are treated as different operational views of one coordinated system rather than as separate software stages (Liang et al., 17 Mar 2025, Wang et al., 20 Feb 2026, Li et al., 2 Apr 2026).