Unified Perception in Multi-Task Systems

• Unified Perception is a framework that integrates tasks like detection, tracking, and prediction into one module while retaining interpretable, explicit outputs.
• It employs shared representations—from common encoders to token-based interfaces—to fuse heterogeneous modalities and enable robust cross-task interactions.
• Empirical studies in autonomous driving, vision-language systems, and medical analysis demonstrate its potential to boost performance and efficiency.

Unified perception denotes a family of methods that integrate tasks, modalities, temporal context, or reasoning stages that are ordinarily separated into distinct modules. In autonomous vehicle software, it is defined as the joint integration of detection, tracking, and prediction inside a single perception module (Stratil et al., 28 Aug 2025). In fine-grained vision-language systems, it appears as a single open-ended language interface spanning object-level detection, pixel-level segmentation, and image-level vision-language tasks (Tang et al., 3 Mar 2025). In 3D CT analysis, it is formulated as a perception-to-understanding loop that accumulates intermediate evidence in long-term structured memory (Wang et al., 20 Feb 2026). In air-ground simulation, it means that aerial and ground agents share one dynamically configurable 3D world and produce temporally and spatially aligned RGB, depth, and semantic segmentation observations from that common state (Wang et al., 17 Oct 2025). This suggests that unified perception is best understood as a systems principle: heterogeneous perceptual functions are made to share a representation, an interface, or an optimization process while preserving task-relevant outputs.

1. Conceptual scope and distinguishing features

Unified perception is not synonymous with a single architectural recipe. Some formulations use a shared encoder with task-specific decoders, as in NeurAll’s definition of a unified visual perception model as a single CNN with a shared encoder and task-specific decoders for object recognition, semantic segmentation, motion estimation, depth estimation, and localization (Sistu et al., 2019). Others remove task-specific decoders almost entirely: UFO transforms detection, segmentation, and vision-language targets into the language space and uses a single <MASK> token embedding to retrieve segmentation masks from image features (Tang et al., 3 Mar 2025). VisionReasoner instead standardizes heterogeneous tasks through an object-centric instruction-and-response schema, with one model producing structured reasoning and outputs that can be mapped to detection, segmentation, or counting (Liu et al., 17 May 2025).

A central distinction separates unified perception from purely modular pipelines. The survey on autonomous vehicle software characterizes modular perception as detection $\rightarrow$ tracking $\rightarrow$ prediction, with error accumulation and limited inter-task synergy at module boundaries, whereas unified perception preserves interpretable outputs while enabling shared optimization and contextual interaction across tasks (Stratil et al., 28 Aug 2025). This is also visible in the depth-aware video panoptic segmentation framework titled "Unified Perception" (Stolle et al., 2023), where panoptic segmentation, monocular depth estimation, and object tracking are handled within one end-to-end framework, and tracking reuses object embeddings already computed by the image-based network.

Unified perception is also not equivalent to end-to-end planning. The survey explicitly positions it between modular interpretability and end-to-end holism: explicit outputs such as boxes, tracks, trajectories, occupancy, masks, or semantic maps are retained even when their generation is jointly optimized (Stratil et al., 28 Aug 2025). A plausible implication is that the term identifies a design objective—shared perception without discarding auditable intermediate representations—rather than a commitment to any one supervision regime or model class.

2. Structural paradigms and common interfaces

A recurring pattern is task integration through a shared intermediate space. In autonomous driving, the survey proposes three paradigms—Early Unified Perception, Late Unified Perception, and Full Unified Perception—corresponding to joint detection+tracking, tracking+prediction, and detection+tracking+prediction, respectively (Stratil et al., 28 Aug 2025). It further distinguishes explicit and implicit tracking, post-network association and in-network association, and representation flows such as camera-to-BEV lifting, LiDAR voxelization, recurrent memory, and message passing.

Other domains organize unification around different interfaces:

Domain	Unified target	Representative formulation
Autonomous vehicle software	Detection, tracking, prediction	EUP, LUP, FUP (Stratil et al., 28 Aug 2025)
Vision-language fine-grained perception	Detection, segmentation, image-level tasks	Open-ended language interface with <MASK> retrieval (Tang et al., 3 Mar 2025)
Collaborative perception	Space-time fusion or communication-collaboration-reconstruction	Unified spatiotemporal space (Tang et al., 1 Aug 2025); multi-resolution MCP (Wang et al., 2023)
Vision-centric 3D driving perception	Occupancy prediction and 3D object detection	Join–divide–join framework (Hong et al., 2024)
LiDAR perception	Detection, semantic segmentation, panoptic segmentation	Shared sparse 3D backbone with GCP (Ye et al., 2022)
Medical analysis	Segmentation, localization, measurement, clinical reasoning	Evidence-driven memory loop (Wang et al., 20 Feb 2026)
Air-ground simulation	Synchronized aerial-ground multi-modal observations	Shared 3D world and timeline (Wang et al., 17 Oct 2025)

The interface itself often becomes the unifying mechanism. UFO encodes boxes as text inside <box>...</box> and masks through embedding retrieval (Tang et al., 3 Mar 2025). VisionReasoner emits a list of object entries of the form {'bbox_2d': [x1, y1, x2, y2], 'point_2d': [cx, cy]}, from which detection returns $(B_i)_{i=1}^N$, segmentation returns $(M_i)_{i=1}^N$, and counting returns $N$ (Liu et al., 17 May 2025). In the Perception-Language-Action framework for autonomous driving, multimodal sensing is fused into a structured text scene description relative to the ego vehicle, which is then consumed by a GPT-4.1 reasoning core to produce action plans and explanations (Zhang et al., 31 Jul 2025).

These examples indicate that “unified” may refer either to shared latent features, to shared output schemas, or to shared procedural loops. This suggests that interface design is as central to unification as backbone design.

3. Representations, fusion mechanisms, and synchronization

Shared backbones with specialized heads remain a dominant pattern. NeurAll uses a shared encoder with multiple decoders and demonstrates that a two-stream three-task network can improve semantic segmentation while keeping parameter count nearly constant at 23.87M for the three-task model (Sistu et al., 2019). LidarMultiNet uses a strong 3D voxel-based encoder–decoder, a Global Context Pooling module that converts sparse 3D features into dense multi-scale BEV and back, and task-specific heads for 3D detection, semantic segmentation, and panoptic segmentation (Ye et al., 2022). UniVision adopts a join–divide–join structure: an explicit–implicit view transform lifts multi-view images into voxel features, local voxel and global BEV branches process complementary structure, and neighborhood cross-attention exchanges information between representations before occupancy and detection heads operate on fused features (Hong et al., 2024).

A second motif is joint spatiotemporal or multi-agent fusion. CoST rejects the conventional decomposition of multi-agent fusion followed by temporal fusion, and instead registers all observations into a unified BEV spatiotemporal space. Its Spatio-temporal Transmission module computes

$$E_i^{t} = \operatorname{Sigmoid}(\phi(\mathbf{F}_i^{t})), \qquad D_i^{t} = \left| E_i^{t} - E_i^{t-\tau} \right|,$$

then forms

$$M_i^{t} = E_i^{t} \cdot \left( \frac{1}{\rho + 1} + D_i^{t} \cdot \rho \right), \qquad z_i^{t} = \mathbf{F}_i^{t} \odot M_i^{t},$$

so that static content is transmitted once and reused from memory while dynamic content is selectively updated (Tang et al., 1 Aug 2025). UMC treats communication, collaboration, and reconstruction as one multi-resolution system: trainable query matrices select informative regions, graph-based G-CGRU fuses them across agents and time, and MGFE reconstructs task-ready features (Wang et al., 2023).

A third motif is synchronization by construction. TranSimHub places aerial and ground agents in one shared 3D world governed by a single simulation timeline, with synchronized RGB, depth, and semantic segmentation outputs generated for heterogeneous cameras attached to UAVs, vehicles, or infrastructure (Wang et al., 17 Oct 2025). Its calibration model is standard:

$x = K [R \mid t] X,$

with SE(3) composition

$T_{ag} = T_a T_g^{-1},$

allowing alignment between aerial and ground frames. The paper proposes a conceptual synchronization metric,

$$E_{\text{sync}} = \| t_{\text{air}} - t_{\text{ground}} \|,$$

although no measured $\rightarrow$0 values are reported (Wang et al., 17 Oct 2025).

Language-centered formulations use the token space itself as the shared representation. UFO retrieves masks from the <MASK> token embedding by

$\rightarrow$1

and upsamples masks using multiple mask tokens and a reshape operation over their similarity maps (Tang et al., 3 Mar 2025). VisionReasoner similarly normalizes heterogeneous tasks through object-centric primitives and a single interface,

$\rightarrow$2

then derives detection, segmentation, or counting outputs from the same response format (Liu et al., 17 May 2025).

4. Optimization strategies and evaluation protocols

Because unified perception couples objectives that were historically trained separately, loss design becomes central. The AV survey presents the generic multi-task objective

$\rightarrow$3

and also reports uncertainty-based weighting,

$\rightarrow$4

as a common way to adapt task weights (Stratil et al., 28 Aug 2025). NeurAll shows that naive sums can be poor scalarizations: in its two-task and three-task experiments, geometric-mean objectives outperformed sum-of-losses baselines, and the three-task model with product/geometric mean reached 82.20% segmentation on KITTI while maintaining depth and motion close to single-task levels (Sistu et al., 2019). UniVision uses progressive loss weight adjustment,

$\rightarrow$5

with $\rightarrow$6 increasing across epochs because direct summation often failed to converge in early training (Hong et al., 2024).

Reinforcement learning provides a different unification strategy when outputs are normalized into a common schema. VisionReasoner trains with GRPO and combines format rewards with matched localization rewards after Hungarian assignment. Its objective is written as

$\rightarrow$7

with accuracy rewards based on IoU and L1 thresholds for matched boxes and points (Liu et al., 17 May 2025). In this case, unification comes not from a shared decoder head but from a shared reward design over detection, segmentation, and counting.

Evaluation is fragmented across domains. Autonomous driving papers report mAP, NDS, MOTA, HOTA, IDF1, ADE, FDE, NLL, and mIoU (Stratil et al., 28 Aug 2025). Depth-aware video panoptic segmentation uses DVPQ and STQ; the "Unified Perception" DVPS framework reports DVPQ 57.1 on Cityscapes-DVPS and STQ 59.1 on KITTI-STEP (Stolle et al., 2023). Collaborative perception introduces collaboration-specific metrics such as ARSV, ARCV, ARCI, and ARTC because AP alone cannot isolate the gains due to shared visibility and temporal continuity (Wang et al., 2023). Medical unified perception-to-understanding is evaluated with MCQ accuracy across Recognition, Visual Reasoning, and Medical Reasoning subtypes (Wang et al., 20 Feb 2026). This suggests that unified perception lacks a universal benchmark because the term spans different kinds of unification.

5. Major application domains and empirical record

Autonomous driving is the most developed domain for unified perception. LidarMultiNet unifies 3D object detection, semantic segmentation, and panoptic segmentation in a single LiDAR-only model and reports 71.13 mIoU on the Waymo 3D semantic segmentation challenge 2022 test set, 76.35 mAPH L2 on the Waymo 3D object detection test set, and on nuScenes test set 81.4 mIoU for segmentation, 67.0 mAP and 71.6 NDS for detection, and 81.4 PQ for panoptic segmentation (Ye et al., 2022). UniVision unifies occupancy prediction and 3D object detection and reports +1.5 mIoU on nuScenes LiDAR segmentation, +1.8 NDS on nuScenes detection, +1.5 mIoU on OpenOccupancy, and +1.8 mIoU on Occ3D over prior state of the art (Hong et al., 2024). The survey on AV software generalizes these efforts into a single design space and explicitly argues that unified perception can retain interpretable outputs while improving robustness and efficiency (Stratil et al., 28 Aug 2025).

Video-centric and depth-aware scene understanding provides another strand. The framework explicitly titled "Unified Perception" achieves depth-aware video panoptic segmentation without video-based training by reusing instance embeddings from an image-based panoptic network and associating them with a two-stage tracker. It reports an overall DVPQ of 57.1 on Cityscapes-DVPS and STQ of 59.1 on KITTI-STEP, exceeding state-of-the-art methods that employ the same backbone network (Stolle et al., 2023). This directly contradicts the common assumption that unified video perception necessarily requires video-based supervision.

Collaborative perception extends unification across agents and time. CoST reports [email protected]/[email protected] of 0.710/0.440 on V2V4Real, 0.915/0.814 on V2XSet, and 0.614/0.473 on DAIR-V2X, while also reducing communication through selective transmission in a unified spatiotemporal space (Tang et al., 1 Aug 2025). UMC frames the problem even more broadly as joint optimization of communication, collaboration, and reconstruction; on V2X-Sim it reports AP 67.80/60.01 at IoU 0.5/0.7 and the best ARCV values 67.01/60.04, indicating strong recovery of objects visible to collaborators but not to the ego agent (Wang et al., 2023).

Vision-language and reasoning-oriented formulations push unified perception beyond classical perception stacks. UFO reports that after multi-task training on five standard visual perception datasets it outperforms previous state-of-the-art generalist models by 12.3 mAP on COCO instance segmentation and 3.3 mIoU on ADE20K semantic segmentation, while also integrating with existing MLLMs for reasoning segmentation (Tang et al., 3 Mar 2025). VisionReasoner evaluates ten tasks across detection, segmentation, and counting and reports relative gains over Qwen2.5-VL-7B of 29.1% on COCO detection, 22.1% on ReasonSeg segmentation, and 15.3% on CountBench counting (Liu et al., 17 May 2025). These results indicate that a unified output schema can be competitive even when tasks traditionally rely on separate heads and losses.

Unified perception also appears where perception is coupled to downstream reasoning or control. In 3D medical analysis, 3DMedAgent turns CT interpretation into an evidence acquisition loop using VISTA3D, CT-CLIP, and a long-term structured memory. It reports total average accuracy 0.66 on DeepTumorVQA and 0.57 on DeepChestVQA with a GPT-5 agent, outperforming general, medical, and 3D-specific MLLMs across more than 40 tasks (Wang et al., 20 Feb 2026). In adaptive autonomous driving, the PLA framework fuses cameras, LiDAR, and radar into a structured scene description, uses GPT-4.1 for risk assessment and one-second planning, and reports speed prediction MAE 0.39 m/s with $\rightarrow$8 0.923, steering angle MAE 2.52° with $\rightarrow$9 0.537, and trajectory ADE 1.013 m and FDE 2.026 m in a construction-zone intersection scenario (Zhang et al., 31 Jul 2025). TranSimHub does not present quantitative fusion benchmarks, but it contributes infrastructure: synchronized multi-view RGB, depth, and semantic segmentation, communication modeling through GNS3 and WinProp, and a causal scene editor for counterfactual analysis across weather, emergency events, and dynamic obstacles (Wang et al., 17 Oct 2025).

6. Limitations, misconceptions, and open research directions

A persistent misconception is that unification automatically improves all constituent tasks. Several papers explicitly document the need for careful balancing. NeurAll shows that naive sum-of-losses baselines can underperform badly relative to geometric-mean scalarization (Sistu et al., 2019). UniVision reports that training can fail without progressive loss adjustment (Hong et al., 2024). The AV survey lists negative transfer, training stability, and multi-task balance as open challenges for unified perception systems (Stratil et al., 28 Aug 2025).

A second misconception is that unified perception is always more interpretable. It often preserves explicit outputs, but the basis of those outputs may shift from classical module boundaries to latent memories, queries, or language tokens. UFO’s masks are retrieved from token embeddings rather than generated by a dedicated mask decoder (Tang et al., 3 Mar 2025); VisionReasoner’s detections, masks, and counts depend on structured reasoning traces and reward-shaped object matching (Liu et al., 17 May 2025). This suggests that interpretability is preserved at the interface level, not necessarily at the mechanistic level.

Systems questions remain underreported in several formulations. TranSimHub defines synchronized air-ground perception but does not report measured synchronization errors, frame jitter, or determinism guarantees under varied loads (Wang et al., 17 Oct 2025). The PLA driving framework does not specify a formal MPC objective, a vehicle model, or hardware latency (Zhang et al., 31 Jul 2025). VisionReasoner notes that LVLM detection lacks native confidence scores and that structured reasoning increases latency (Liu et al., 17 May 2025). 3DMedAgent depends on the reliability of VISTA3D and CT-CLIP, so tool errors can propagate into memory and reasoning (Wang et al., 20 Feb 2026).

Future directions recur across the literature. The survey emphasizes long-horizon temporal reasoning, stronger memory architectures, better multi-modal fusion, principled multi-task balancing, and reproducible open-source baselines for LUP and FUP systems (Stratil et al., 28 Aug 2025). TranSimHub identifies sensor realism, communication fidelity, aerial dynamics, and large-scale benchmarks as unresolved (Wang et al., 17 Oct 2025). VisionReasoner points to richer confidence modeling, more diverse RL data, and extension to tracking and pose (Liu et al., 17 May 2025). 3DMedAgent identifies learned routing, improved tool-use proficiency, broader modality coverage, and clinical validation under human supervision as natural next steps (Wang et al., 20 Feb 2026).

Taken together, the literature portrays unified perception as an increasingly general organizing principle for machine perception. Its concrete realizations differ—shared encoders, BEV-voxel hybrids, language interfaces, object queries, spatiotemporal fusion spaces, or evidence-driven agent loops—but they converge on the same claim: perceptual competence improves when related tasks, modalities, and temporal contexts are allowed to share structure rather than communicate only through brittle module boundaries.