CoST: Spatio-temporal Collaborative Perception

• The paper introduces a unified spatio-temporal fusion where historical observations act as time-delayed copies to improve cross-agent and cross-time interactions.
• It employs a Transformer-style module with Multi-Agent Deformable Attention and Align Transformation to efficiently fuse features and minimize redundant transmissions.
• Empirical results demonstrate significant accuracy gains and up to 100× bandwidth savings on benchmarks like V2V4Real, V2XSet, and DAIR-V2X with minimal performance loss.

Collaborative Perception with Spatio-temporal Transformer (CoST) is a LiDAR-based cooperative 3D perception framework that reformulates collaborative perception from a unified spatio-temporal perspective rather than the conventional decomposition into separate multi-agent and multi-time fusion stages. Its central premise is that historical observations can be treated as time-delayed copies of current agents, so that features from different agents and different timestamps can be embedded into a single aggregation space and fused in one Transformer-style operation. This design is used to pursue two linked objectives: more holistic cross-agent, cross-time interaction modeling and more efficient communication through selective transmission of dynamic content rather than repeated transmission of static regions (Tang et al., 1 Aug 2025).

1. Conceptual basis

CoST addresses the “where” and “when” dimensions of collaborative perception jointly. The stated objectives are to overcome single-agent blind spots such as occlusions and limited sensing range by sharing information among $N$ agents, to exploit temporal context to correct pose errors, compensate delays, and smooth detections, and to do both simultaneously in a single unified spatio-temporal fusion step rather than two separate stages. The system setup assumes $N$ agents, including vehicles and/or infrastructure, each equipped with LiDAR; at timestamp $t$, agent $i$ extracts a Bird’s-Eye-View feature map

$\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$

and the system maintains a memory bank of reconstructed historical features $\mathbf{R}_i^{t-\tau}$ for each agent (Tang et al., 1 Aug 2025).

The principal conceptual departure from prior two-stage fusion is that historical features are treated as additional observations in the same fusion space, distinguished by timestamp rather than by a separate temporal-processing pipeline. The paper contrasts this with prior methods that first fuse agents in space and then fuse over time, or invert that order. In CoST, all $\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$ are embedded into one spatio-temporal space and fused in a single module. A common misunderstanding is to equate this viewpoint with mere temporal stacking; the paper’s formulation is more specific, because it ties the unified representation directly to a communication policy that avoids re-transmitting static object features.

2. Unified spatio-temporal representation

The unified representation is defined over a spatial grid and a temporal set. CoST introduces

$S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$

and

$T=\{t-\tau,\,t\},$

with the note that in practice the method can use up to $L$ historical frames, although recurrent modeling keeps only the last fused feature. The aggregation space is therefore $N$0, and the set of feature maps is

$N$1

These maps are embedded into a joint tensor of tokens,

$N$2

where each spatio-temporal location $N$3 is linearly projected into a $N$4-dimensional token (Tang et al., 1 Aug 2025).

This representation makes the agent axis and the temporal axis structurally parallel within the fusion module. Every token is indexed simultaneously by agent, timestamp, and BEV position, which allows cross-agent and cross-time interactions to be learned in a single attention mechanism rather than through serial composition. This suggests that the method’s performance gains are not only a consequence of adding temporal information, but also of changing the topology of information exchange so that interactions across time and across agents are resolved in one latent space.

3. Transformer fusion mechanism

CoST’s core fusion module is described as agnostic to the exact attention implementation. The paper gives an efficient instantiation, Multi-Agent Deformable Attention (MADA), together with an Align Transformation (AT) that injects agent metadata. AT augments features with pose latency, velocity, and discrete agent type: $N$5 where $N$6 and $N$7 denote discrete and continuous meta-features, and $N$8 (Tang et al., 1 Aug 2025).

The deformable attention operator is then written as

$N$9

Here, for each query token $t$0, the model samples $t$1 offsets $t$2 from each agent’s feature map $t$3. The previous-frame fused feature $t$4, after pose-warp, is treated as an additional “agent,” so the same block simultaneously fuses current ego features, other agents’ features, and historical context. Positional information is introduced through learnable spatial embeddings for $t$5, a fixed time-delay embedding for historical tokens at $t$6, and agent identity through AT. The paper states that every query token can attend to every other token across both agent and timestamp, while deformable sampling restricts computation to sparse, high-score regions, which is the mechanism used to retain efficiency (Tang et al., 1 Aug 2025).

A second common misunderstanding is to regard CoST as inseparable from MADA. The paper explicitly states the opposite: the fusion module is agnostic to the exact attention implementation, and MADA is presented as one efficient instantiation rather than as the definition of CoST itself.

4. Spatio-temporal transmission

CoST’s communication scheme is the Spatio-temporal Transmission module (STT). Its starting point is the observation that intermediate-fusion methods often compress and transmit entire BEV maps, or select salient tokens per frame, but still re-send static regions at each step. STT instead uses temporal redundancy to transmit only dynamic and salient tokens. The saliency map is

$t$7

where $t$8 is the detection-head classification branch. The motion-based dynamic map is

$t$9

The selection mask and transmitted feature are

$i$0

Static tokens are reconstructed from $i$1 using a small CNN (Tang et al., 1 Aug 2025).

The bandwidth model is correspondingly altered. The baseline transmits full features of size $i$2, whereas CoST transmits only

$i$3

The paper states that this is often $i$4–$i$5 of the full map and reports up to $i$6–$i$7 bandwidth savings with negligible AP loss; in the bandwidth–accuracy analysis, CoST-STT reduces bandwidth by up to $i$8 with less than $i$9 AP drop (Tang et al., 1 Aug 2025).

The significance of STT is not merely compression. It is a communication rule derived from the unified spatio-temporal model: because static structure can be recovered from the historical state, repeated transmission of those regions becomes unnecessary. That distinguishes STT from frame-local sparsification schemes that reduce payload size but do not exploit inter-frame redundancy in the transmission policy itself.

5. Empirical performance and ablation structure

CoST is evaluated on V2V4Real, V2XSet, and DAIR-V2X, using PointPillar backbones and AP at IoU thresholds $\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$0. The headline comparisons reported in the paper are as follows (Tang et al., 1 Aug 2025).

Benchmark	CoST [email protected] / [email protected]	Best prior [email protected] / [email protected]
V2V4Real	0.710 / 0.440	0.662 / 0.379
V2XSet	0.915 / 0.814	0.881 / 0.771
DAIR-V2X	0.614 / 0.473	0.608 / 0.466

These values correspond to gains of $\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$1 on V2V4Real, $\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$2 on V2XSet, and $\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$3 on DAIR-V2X. The robustness analysis further reports that under Gaussian pose errors and transmission delays from $\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$4 to $\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$5 ms, CoST maintains significantly higher AP than the baselines.

The ablation structure is organized around three components: MADA, AT, and the unified spatio-temporal fusion component denoted USTF. On V2V4Real, MADA alone yields $\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$6; adding AT yields $\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$7, a gain of $\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$8; adding USTF yields $\mathbf{F}_i^t \in \mathbb{R}^{H\times W\times C},$9, a gain of $\mathbf{R}_i^{t-\tau}$0; adding both yields $\mathbf{R}_i^{t-\tau}$1, a gain of $\mathbf{R}_i^{t-\tau}$2. The temporal ablation reports $\mathbf{R}_i^{t-\tau}$3 for $\mathbf{R}_i^{t-\tau}$4 frame, $\mathbf{R}_i^{t-\tau}$5 for $\mathbf{R}_i^{t-\tau}$6 frames with interval $\mathbf{R}_i^{t-\tau}$7, $\mathbf{R}_i^{t-\tau}$8 for $\mathbf{R}_i^{t-\tau}$9 frames with interval $\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$0, and $\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$1 for $\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$2 frames with interval $\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$3. The paper’s stated interpretation is that agent metadata and temporal history both add value and that there is diminishing return beyond $\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$4–$\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$5 frames at interval $\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$6.

6. Compatibility and relation to adjacent spatio-temporal V2X fusion

CoST is presented as method-agnostic at the integration level. STT is described as a drop-in replacement for any feature-compression module: down-/up-sampling convolutions are replaced with the selective-token mechanism while the original fusion remains unchanged. USTF is described as wrapping around any multi-agent fusion block: after extracting per-agent BEV features, the method pose-warps and tags the last fused BEV as an extra “historical agent,” then feeds the resulting $\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$7 features into the existing fusion block. The paper further states that integrating STT+USTF into V2X-ViT and CoBEVT yields both reduced bandwidth and AP gains (Tang et al., 1 Aug 2025).

A useful comparison is V2XPnP, which likewise centers spatio-temporal fusion in V2X scenarios but targets end-to-end perception and prediction rather than collaborative perception alone. V2XPnP studies one-step and multi-step communication strategies, examines their integration with early, late, and intermediate fusion, provides benchmarks with $\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$8 fusion models, and proposes an intermediate-fusion framework within one-step communication. Its architecture uses a temporal Transformer for per-agent historical fusion, a spatial Transformer for multi-agent fusion, and HD-map injection before decoding into a 3D-object detection head and a trajectory-prediction head (Zhou et al., 2024).

The contrast is technically informative. CoST collapses multi-agent and multi-time aggregation into a single spatio-temporal space and derives an STT policy that transmits only dynamic features. V2XPnP, by contrast, retains an explicit stage structure: metadata sharing, LiDAR feature extraction, per-agent temporal fusion, one-step sharing, multi-agent spatial fusion, and map injection. It also reports that one-step communication in vehicle-centric mode achieves AP$\{\mathbf{F}_i^t,\mathbf{R}_i^{t-\tau}\}$9, EPA$S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$0, compared with AP$S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$1, EPA$S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$2 for multi-step communication, and under $S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$3 ms delay plus $S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$4 noise the one-step scheme incurs $S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$5 EPA versus $S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$6 for multi-step (Zhou et al., 2024). This comparison suggests a broader methodological shift in V2X research toward communication policies that are explicitly co-designed with spatio-temporal fusion rather than appended to frame-wise cooperative perception after the fact.

7. Limitations and research directions

The limitations stated for CoST are principally computational, task-related, and temporal-modeling related. Storing historical fused features and running cross-time attention adds modest overhead; Table 5 reports inference time $S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$7 ms versus $S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$8 ms for CoST-S. The paper notes that scaling to very large agent fleets or extremely long temporal windows may require further efficiency gains. It also limits its experiments to LiDAR-based 3D detection and identifies semantic segmentation and camera-based BEV as possible extensions. A further issue is dynamic agent count: frequent agent join/leave events may complicate the recurrent historical fusion, motivating mechanisms to forget stale agents or adaptively adjust memory. Finally, CoST fixes the temporal delay $S=\{(x,y) : x=1\ldots H,\;y=1\ldots W\}$9, and the paper identifies learned time alignment and more flexible handling of asynchrony as open directions (Tang et al., 1 Aug 2025).

These limitations delimit the scope of the current formulation rather than negate its contribution. CoST’s specific contribution lies in unifying spatio-temporal aggregation and communication design: the method does not merely add temporal context to collaborative perception, but redefines transmission, memory, and fusion around a single spatio-temporal representation. Within the recent literature on V2X spatio-temporal modeling, that makes it a reference point for research that seeks to optimize both perception accuracy and communication efficiency under realistic latency and bandwidth constraints.