Robust Perception: Trade-offs and Methods

• Robust perception is a systems-level approach that ensures reliable estimation despite sensor perturbations, distribution shifts, and hardware faults.
• It employs techniques such as adversarial training, uncertainty propagation, output consensus, and inference-time adaptation to enhance resilience.
• Applications span autonomous driving, robotics, and multi-agent systems, where maintaining performance under real-world disturbances is critical.

Searching arXiv for the cited robust perception papers to ground the article in current literature. Robust perception denotes the design of perceptual systems that remain reliable when sensing, inference, or deployment conditions deviate from nominal assumptions. In the cited literature, the term spans several technically distinct settings: adversarially robust collaborative perception in multi-agent systems (Li et al., 2023), robust perception-based control under learned sensing error (Makdah et al., 2019, Dean et al., 2019, Jarin-Lipschitz et al., 2020), uncertainty-aware localization and mapping (Sirohi et al., 2024), architectural co-design for dependable automotive perception (Dey et al., 2022), robustness to missing views in BEV pipelines (Chen et al., 2023), inference-time robustness mechanisms based on equivariance or canonicalization (Mao et al., 2022, Singhal et al., 14 Jul 2025), certification against camera motion perturbations (Hu et al., 2022), robustness to asynchronous collaboration and sensor misalignment (Xu et al., 12 Feb 2025, Xia et al., 2024), and robust articulated-object perception for manipulation (Wang et al., 2024). Taken together, these works define robust perception not as a single algorithmic recipe but as a systems-level objective: preserving perceptual validity under perturbation, distribution shift, uncertainty, hardware faults, temporal misalignment, or perception–control coupling.

1. Scope and formal problem settings

A recurring formulation treats perception as an estimator embedded in a larger downstream system. In perception-based control, the learned perception map $\phi:\mathbb{R}^{p\times q}\to\mathbb{R}^m$ is trained from a finite dataset $D_N=\{(Z_i,x_i)\}_{i=1}^N$ by minimizing mean squared error, after which it is linearized around nominal operation as $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$ with $v(t)\sim\mathcal{N}(0,R)$ (Makdah et al., 2019). Robustness is then defined not only by nominal estimation performance $\mathcal{P}(K):=\mathrm{Tr}\,P(K)$, but also by sensitivity of the steady-state error covariance to perturbations in the learned noise model, $S(K):=\mathrm{Tr}[dP(K)/dR]$ (Makdah et al., 2019). The central claim is that algorithms maximizing nominal estimation accuracy tend to perform poorly when sensor statistics differ from the learned ones, whereas increasing training variability improves robustness while limiting nominal performance (Makdah et al., 2019).

A related formalism appears in robust guarantees for perception-based control, where a learned perception map $\hat h:\mathbb{R}^M\to\mathbb{R}^p$ approximates a linear function of state, $\hat y_t=\hat h(z_t)\approx Cx_t$, with worst-case observation error

$$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$

over an operating region $X_{\mathrm{op}}$ (Dean et al., 2019). Here robustness is tied to certifiable safe-set construction and invariance of the closed loop under bounded perception error. The same theme is carried into quadrotor control, where VIO bias is modeled as $D_N=\{(Z_i,x_i)\}_{i=1}^N$0 and robust synthesis explicitly constrains the closed-loop gain from perception error to state (Jarin-Lipschitz et al., 2020).

Other works pose robust perception directly at the perception layer. In collaborative perception, robustness is defined against adversarial teammates or temporal asynchrony. ROBOSAC assumes that collaborative perception should lead to consensus rather than dissensus relative to ego-only perception, and accepts fusion only when repeated random subset sampling yields output-space agreement (Li et al., 2023). CoDynTrust models temporally asynchronous collaboration as a feature-quality problem and introduces a dynamic feature trust modulus based on aleatoric and epistemic uncertainty (Xu et al., 12 Feb 2025). In BEV perception, M-BEV formulates robustness as resilience to one or more failed cameras by randomly masking and reconstructing view features during training (Chen et al., 2023).

A different line of work defines robust perception through invariance or equivariance. FoCal seeks a canonicalizer $D_N=\{(Z_i,x_i)\}_{i=1}^N$1 such that, for transformed input $D_N=\{(Z_i,x_i)\}_{i=1}^N$2, a canonical transform $D_N=\{(Z_i,x_i)\}_{i=1}^N$3 maps it toward a visually typical view by minimizing a foundation-model energy (Singhal et al., 14 Jul 2025). The equivariance-based framework instead solves an inference-time constrained optimization over an $D_N=\{(Z_i,x_i)\}_{i=1}^N$4 ball around the attacked input to restore feature equivariance under a transformation group $D_N=\{(Z_i,x_i)\}_{i=1}^N$5 (Mao et al., 2022). These formulations suggest that robust perception can be approached either by hardening the estimator, constraining inference, or redesigning the interface between perception and downstream decision-making.

2. Robustness mechanisms in collaborative and multi-agent perception

Collaborative perception introduces vulnerabilities absent from single-agent systems because information exchange can amplify corrupted, delayed, or malicious signals. ROBOSAC addresses adversarial feature-map perturbations in collaborative 3D object detection by replacing trust-all fusion with a hypothesize-and-verify loop (Li et al., 2023). At each perception step, the ego robot computes a solo forward pass $D_N=\{(Z_i,x_i)\}_{i=1}^N$6, then samples $D_N=\{(Z_i,x_i)\}_{i=1}^N$7 out of $D_N=\{(Z_i,x_i)\}_{i=1}^N$8 teammates uniformly at random, computes $D_N=\{(Z_i,x_i)\}_{i=1}^N$9, and accepts collaboration only if a difference measure $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$0 (Li et al., 2023). In 3D object detection, the outputs are sets of axis-aligned 3D boxes; after one-to-one matching by the Hungarian algorithm, the paper defines

$y(t):=\phi(Z(t))\approx Cx(t)+v(t)$1

and treats consensus as an output-space criterion rather than a feature-space norm test (Li et al., 2023).

ROBOSAC also provides closed-form sampling bounds. If $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$2 is the fraction of attackers among $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$3 peers, then a sampled $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$4-subset is attacker-free with probability $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$5, and after $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$6 independent samplings the probability that at least one draw is clean is

$y(t):=\phi(Z(t))\approx Cx(t)+v(t)$7

Solving yields

$y(t):=\phi(Z(t))\approx Cx(t)+v(t)$8

(Li et al., 2023). On V2X-Sim with white-box PGD attacks, $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$9, target $v(t)\sim\mathcal{N}(0,R)$0, and success probability $v(t)\sim\mathcal{N}(0,R)$1, the equations give $v(t)\sim\mathcal{N}(0,R)$2; under that setting ROBOSAC achieves [email protected]/[email protected] of $v(t)\sim\mathcal{N}(0,R)$3 versus $v(t)\sim\mathcal{N}(0,R)$4 for no defense and $v(t)\sim\mathcal{N}(0,R)$5 for ego only (Li et al., 2023). Under C&W black-box attacks never seen by a PGD-trained model, PGD-based adversarial training drops to $v(t)\sim\mathcal{N}(0,R)$6, whereas ROBOSAC retains $v(t)\sim\mathcal{N}(0,R)$7 (Li et al., 2023). This supports the paper’s attack-agnostic interpretation of robustness as outlier rejection rather than attacker modeling.

Temporal asynchrony is treated differently in CoDynTrust. For each agent and timestamp, CoDynTrust predicts aleatoric uncertainty with a direct-modeling head and epistemic uncertainty with MC-Dropout, rescales raw uncertainties to a common range, and computes a trust modulus per ROI through a small residual-block network $v(t)\sim\mathcal{N}(0,R)$8 on averaged confidence and uncertainty across the two most recent frames (Xu et al., 12 Feb 2025). A delay-decay factor $v(t)\sim\mathcal{N}(0,R)$9 with $\mathcal{P}(K):=\mathrm{Tr}\,P(K)$0 modulates trust according to temporal lag:

$\mathcal{P}(K):=\mathrm{Tr}\,P(K)$1

(Xu et al., 12 Feb 2025). The trust-scaled features are then fused by a multi-scale hybrid module that computes MAXOUT, AVGOUT, spatial reweighting, and channel reweighting (Xu et al., 12 Feb 2025). Across DAIR-V2X, V2XSet, and OPV2V, CoDynTrust is reported to show the smallest performance degradation as delay increases; on DAIR-V2X at [email protected], it achieves $\mathcal{P}(K):=\mathrm{Tr}\,P(K)$2 at $\mathcal{P}(K):=\mathrm{Tr}\,P(K)$3, $\mathcal{P}(K):=\mathrm{Tr}\,P(K)$4 at $\mathcal{P}(K):=\mathrm{Tr}\,P(K)$5, and $\mathcal{P}(K):=\mathrm{Tr}\,P(K)$6 at $\mathcal{P}(K):=\mathrm{Tr}\,P(K)$7, while under $\mathcal{P}(K):=\mathrm{Tr}\,P(K)$8 delay plus Gaussian pose noise $\mathcal{P}(K):=\mathrm{Tr}\,P(K)$9 it reaches $S(K):=\mathrm{Tr}[dP(K)/dR]$0 versus $S(K):=\mathrm{Tr}[dP(K)/dR]$1 for CoBEVFlow (Xu et al., 12 Feb 2025).

A further collaborative-perception challenge is spatial-temporal alignment without external localization and clock signals. A 2024 work proposes aligning agents by recognizing inherent geometric patterns in perceptual data rather than depending on external hardware (Lei et al., 2024). Its abstract states that the key module, FreeAlign, constructs a salient object graph for each agent from detected boxes and uses a graph neural network to identify common subgraphs between agents, leading to accurate relative pose and time; it is validated on real-world and simulated datasets, and the resulting system performs comparably to systems relying on precise localization and clock devices (Lei et al., 2024). However, the accompanying details block explicitly states that the provided document contains no technical details about the system (Lei et al., 2024). This limits any more specific encyclopedic treatment of its architecture or experiments.

3. Perception under uncertainty, distribution shift, and perception–control coupling

Robust perception is often constrained by the fact that the perceptual module is learned from finite data and deployed in conditions with shifted sensor statistics. In the control-theoretic treatment of perception-based control, the robust estimator gain is obtained by solving

$S(K):=\mathrm{Tr}[dP(K)/dR]$2

with $S(K):=\mathrm{Tr}[dP(K)/dR]$3 (Makdah et al., 2019). Under assumptions that $S(K):=\mathrm{Tr}[dP(K)/dR]$4 is stable, $S(K):=\mathrm{Tr}[dP(K)/dR]$5 is detectable, $S(K):=\mathrm{Tr}[dP(K)/dR]$6, and $S(K):=\mathrm{Tr}[dP(K)/dR]$7, the optimal gain is parameterized by a single scalar $S(K):=\mathrm{Tr}[dP(K)/dR]$8 through a Riccati equation for $S(K):=\mathrm{Tr}[dP(K)/dR]$9 and

$\hat h:\mathbb{R}^M\to\mathbb{R}^p$0

(Makdah et al., 2019). The paper proves that $\hat h:\mathbb{R}^M\to\mathbb{R}^p$1 is strictly decreasing in $\hat h:\mathbb{R}^M\to\mathbb{R}^p$2 and that the optimal robustness $\hat h:\mathbb{R}^M\to\mathbb{R}^p$3 is strictly decreasing in $\hat h:\mathbb{R}^M\to\mathbb{R}^p$4; equivalently, increasing nominal accuracy forces a loss in robustness (Makdah et al., 2019). In CARLA with a planar double-integrator and a convolutional perception map trained on clear-weather images, the nominal controller yields $\hat h:\mathbb{R}^M\to\mathbb{R}^p$5 and $\hat h:\mathbb{R}^M\to\mathbb{R}^p$6, while the robust controller yields $\hat h:\mathbb{R}^M\to\mathbb{R}^p$7 and $\hat h:\mathbb{R}^M\to\mathbb{R}^p$8 (Makdah et al., 2019). The result is not merely empirical but formulated as a fundamental trade-off.

Robust guarantees for perception-based control pursue a more explicit certification route. Assuming $\hat h:\mathbb{R}^M\to\mathbb{R}^p$9 and $\hat y_t=\hat h(z_t)\approx Cx_t$0 are Lipschitz and the local slope of the perception error $\hat y_t=\hat h(z_t)\approx Cx_t$1 is bounded near training points, the paper constructs a safe set $\hat y_t=\hat h(z_t)\approx Cx_t$2 as a union of local balls in which the error is bounded by $\hat y_t=\hat h(z_t)\approx Cx_t$3 (Dean et al., 2019). An equivalent view uses a quadratic Lyapunov function $\hat y_t=\hat h(z_t)\approx Cx_t$4 and a $\hat y_t=\hat h(z_t)\approx Cx_t$5-inflated safe set

$\hat y_t=\hat h(z_t)\approx Cx_t$6

then chooses the largest invariant sublevel set under the closed-loop map (Dean et al., 2019). In CARLA, a robust $\hat y_t=\hat h(z_t)\approx Cx_t$7 controller solved via SLS with empirically computed $\hat y_t=\hat h(z_t)\approx Cx_t$8 is reported to remain within $\hat y_t=\hat h(z_t)\approx Cx_t$9 of the training trajectory in all $$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$0 randomized trials, whereas nominal controllers exceed $$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$1 error on average (Dean et al., 2019).

The quadrotor extension makes the same perception–control coupling concrete in hardware. There, VIO output is modeled as $$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$2, with $$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$3 treated as a state-dependent bias characterized by a uniform norm bound and an $$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$4-slope estimate learned from logged state and VIO trajectories (Jarin-Lipschitz et al., 2020). Robust control is synthesized in the SLS framework by minimizing either a nominal quadratic cost or an imitation cost relative to an existing controller, while imposing an explicit robustness constraint on the gain from $$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$5 to $$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$6 (Jarin-Lipschitz et al., 2020). In simulation with degraded perception, the PD controller’s tracking and VIO drift grew substantially, the nominal $$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$7 drifted uncontrollably, and both robust SLS controllers maintained bounded drift with tracking error within approximately $$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$8; on hardware, under weaker lighting and sparser texture, PD error grew to approximately $$\Delta := \sup_{x\in X_{\mathrm{op}}}\|\hat h(g(x))-Cx\|$$9, whereas the robust-imitation controller remained under $X_{\mathrm{op}}$0 (Jarin-Lipschitz et al., 2020). A plausible implication is that robustness in perception cannot always be isolated at the perception stack; it may require joint reasoning about estimator bias, safe operating regions, and downstream feedback gains.

Uncertainty-aware localization and mapping provides a related but downstream-oriented view. uPLAM uses the evidential panoptic CNN EvPSNet to estimate Dirichlet evidence per pixel,

$X_{\mathrm{op}}$1

with class probability $X_{\mathrm{op}}$2 and epistemic uncertainty $X_{\mathrm{op}}$3 (Sirohi et al., 2024). It also computes predictive uncertainty through normalized entropy of the Dirichlet means,

$X_{\mathrm{op}}$4

then propagates these uncertainties through BEV map aggregation and particle-filter localization (Sirohi et al., 2024). In map aggregation, cell-wise evidence is fused by averaging Dirichlet parameters; in localization, semantic and landmark mIoU scores are modified by uncertainty-weighted intersections, and the particle likelihood is sharpened by

$X_{\mathrm{op}}$5

with $X_{\mathrm{op}}$6 (Sirohi et al., 2024). On the Freiburg sequence, evidential fusion yields overall mIoU $X_{\mathrm{op}}$7 and uECE $X_{\mathrm{op}}$8 versus $X_{\mathrm{op}}$9 and $D_N=\{(Z_i,x_i)\}_{i=1}^N$00 for log-odds + softmax (Sirohi et al., 2024). With $D_N=\{(Z_i,x_i)\}_{i=1}^N$01 particles and noisy odometry, the full localization system reduces translational MAE from $D_N=\{(Z_i,x_i)\}_{i=1}^N$02 for the baseline $D_N=\{(Z_i,x_i)\}_{i=1}^N$03 model to $D_N=\{(Z_i,x_i)\}_{i=1}^N$04 after regularization, uncertainty incorporation, and instance matching (Sirohi et al., 2024). This locates robust perception not only in prediction accuracy but in calibrated uncertainty that can be exploited by downstream Bayesian inference.

4. Architectural robustness in autonomous driving perception

Several works treat robust perception as a problem of architecture design under hardware and deployment constraints. PASTA formulates dependable automotive perception as a global co-optimization over sensor selection, placement, and orientation; deep-learning object detector choice and parameters; and fusion algorithm choice (Dey et al., 2022). The objective trades cumulative perception loss against hardware cost under constraints on allowable mounting zones, orientation bounds, and maximum number of sensors (Dey et al., 2022). The framework decodes each design point into CARLA simulations, measures eight ADAS metrics, and updates the design with a population-based optimizer (Dey et al., 2022). It supports GA, DE, and FA; for GA, the reported settings are roulette-wheel selection, single-point crossover with rate $D_N=\{(Z_i,x_i)\}_{i=1}^N$05, mutation rate $D_N=\{(Z_i,x_i)\}_{i=1}^N$06, and population size $D_N=\{(Z_i,x_i)\}_{i=1}^N$07 (Dey et al., 2022).

The importance of integrated design is made explicit in the BMW-Minicooper case study. After approximately $D_N=\{(Z_i,x_i)\}_{i=1}^N$08 of search, GA-PASTA, which jointly optimizes position, orientation, detector, and fusion, reaches a best average cost of approximately $D_N=\{(Z_i,x_i)\}_{i=1}^N$09, compared with approximately $D_N=\{(Z_i,x_i)\}_{i=1}^N$10 for GA-PO, which only optimizes position with fixed YOLOv3+EKF (Dey et al., 2022). Among exploration algorithms, FA-PASTA improves on DE by $D_N=\{(Z_i,x_i)\}_{i=1}^N$11 and on GA by $D_N=\{(Z_i,x_i)\}_{i=1}^N$12 for the Audi-TT, and beats DE by $D_N=\{(Z_i,x_i)\}_{i=1}^N$13 and GA by $D_N=\{(Z_i,x_i)\}_{i=1}^N$14 for the BMW (Dey et al., 2022). When neural architecture search is added, FA-NAS-PASTA improves best cost by up to $D_N=\{(Z_i,x_i)\}_{i=1}^N$15 on Audi-TT and $D_N=\{(Z_i,x_i)\}_{i=1}^N$16 on BMW-Minicooper over plain FA-PASTA (Dey et al., 2022). The paper’s design guidelines emphasize that global co-optimization always outperforms sequential or partial searches, and that vehicle-specific geometry alters optimal sensor zones, making one-size-fits-all sensor packs sub-optimal (Dey et al., 2022).

Robustness to sensor failure inside BEV pipelines is addressed by M-BEV. Its Masked View Reconstruction module is inserted after the 2D encoder and before BEV translation, where Random View Masking zeroes out entire feature maps of randomly selected camera views and MVR reconstructs them from the remaining views (Chen et al., 2023). At each training iteration, the masking stage chooses $D_N=\{(Z_i,x_i)\}_{i=1}^N$17 camera views uniformly across subsets, allowing the model to see many missing-camera configurations (Chen et al., 2023). Reconstruction is trained by

$D_N=\{(Z_i,x_i)\}_{i=1}^N$18

and the total objective is $D_N=\{(Z_i,x_i)\}_{i=1}^N$19 with $D_N=\{(Z_i,x_i)\}_{i=1}^N$20 (Chen et al., 2023).

On nuScenes with PETRv2, the full-view baseline gives NDS $D_N=\{(Z_i,x_i)\}_{i=1}^N$21 and mAP $D_N=\{(Z_i,x_i)\}_{i=1}^N$22; removing the back camera reduces PETRv2 to NDS $D_N=\{(Z_i,x_i)\}_{i=1}^N$23 and mAP $D_N=\{(Z_i,x_i)\}_{i=1}^N$24, whereas M-BEV recovers to NDS $D_N=\{(Z_i,x_i)\}_{i=1}^N$25 and mAP $D_N=\{(Z_i,x_i)\}_{i=1}^N$26, a $D_N=\{(Z_i,x_i)\}_{i=1}^N$27 absolute mAP gain over the failed-case baseline (Chen et al., 2023). For random multi-view failure, M-BEV outperforms the baseline by $D_N=\{(Z_i,x_i)\}_{i=1}^N$28–$D_N=\{(Z_i,x_i)\}_{i=1}^N$29 mAP, with local MVR consistently $D_N=\{(Z_i,x_i)\}_{i=1}^N$30–$D_N=\{(Z_i,x_i)\}_{i=1}^N$31 mAP better than global MVR (Chen et al., 2023). In the no-failure case, inference cost is approximately the same as the baseline because the decoder is bypassed, yet the model still gains $D_N=\{(Z_i,x_i)\}_{i=1}^N$32 mAP from robustness-aware training; under single-view failure, Local MVR adds only approximately $D_N=\{(Z_i,x_i)\}_{i=1}^N$33 latency, from $D_N=\{(Z_i,x_i)\}_{i=1}^N$34 to $D_N=\{(Z_i,x_i)\}_{i=1}^N$35 (Chen et al., 2023). This suggests a distinct notion of robust perception: not post hoc defense, but training-time exposure to plausible hardware-failure modes.

Long-range robustness against sensor misalignment is handled by a multi-task LiDAR–camera system that jointly predicts 2D detection, 3D detection, and three-axis rotational misalignment along with calibrated uncertainty (Xia et al., 2024). The misalignment head predicts $D_N=\{(Z_i,x_i)\}_{i=1}^N$36 and diversity parameters $D_N=\{(Z_i,x_i)\}_{i=1}^N$37, trained by a Laplace-NLL-style loss

$D_N=\{(Z_i,x_i)\}_{i=1}^N$38

(Xia et al., 2024). Estimates are fused over a sliding $D_N=\{(Z_i,x_i)\}_{i=1}^N$39 window after rejecting frames whose predicted uncertainty exceeds $D_N=\{(Z_i,x_i)\}_{i=1}^N$40, using inverse-variance weighting (Xia et al., 2024). On the internal long-range dataset with injected perturbations in $D_N=\{(Z_i,x_i)\}_{i=1}^N$41, snippet-level fusion with uncertainty reaches precision $D_N=\{(Z_i,x_i)\}_{i=1}^N$42, recall $D_N=\{(Z_i,x_i)\}_{i=1}^N$43, and mean absolute errors of $D_N=\{(Z_i,x_i)\}_{i=1}^N$44 for roll, $D_N=\{(Z_i,x_i)\}_{i=1}^N$45 for pitch, and $D_N=\{(Z_i,x_i)\}_{i=1}^N$46 for yaw (Xia et al., 2024). In 3D vehicle detection at $D_N=\{(Z_i,x_i)\}_{i=1}^N$47–$D_N=\{(Z_i,x_i)\}_{i=1}^N$48, a CenterNet backbone improves from max-F1 $D_N=\{(Z_i,x_i)\}_{i=1}^N$49 for the miscalibrated baseline to $D_N=\{(Z_i,x_i)\}_{i=1}^N$50 with proposed correction, reported as $D_N=\{(Z_i,x_i)\}_{i=1}^N$51 (Xia et al., 2024). The work explicitly interprets predicted aleatoric uncertainty as useful for temporal filtering and outlier rejection (Xia et al., 2024).

5. Inference-time adaptation, invariance, and certification

A major strand of robust perception shifts the burden of robustness from training to inference. The equivariance-based framework formulates attacked-input restoration as

$D_N=\{(Z_i,x_i)\}_{i=1}^N$52

and predicts with $D_N=\{(Z_i,x_i)\}_{i=1}^N$53 (Mao et al., 2022). The self-supervised objective uses dense feature-space equivariance under a known transformation group $D_N=\{(Z_i,x_i)\}_{i=1}^N$54, measuring per-transform cosine similarity

$D_N=\{(Z_i,x_i)\}_{i=1}^N$55

and minimizing a penalty of the form

$D_N=\{(Z_i,x_i)\}_{i=1}^N$56

(Mao et al., 2022). The paper argues, both theoretically and empirically, that restoring equivariance at inference can reverse adversarial corruption without retraining the model (Mao et al., 2022).

The empirical trade-off is explicit. On ImageNet classification with $D_N=\{(Z_i,x_i)\}_{i=1}^N$57, vanilla inference gives clean accuracy $D_N=\{(Z_i,x_i)\}_{i=1}^N$58 and robust accuracy $D_N=\{(Z_i,x_i)\}_{i=1}^N$59 under PGD-10, while equivariance-based inference yields clean accuracy $D_N=\{(Z_i,x_i)\}_{i=1}^N$60 and robust accuracy $D_N=\{(Z_i,x_i)\}_{i=1}^N$61 (Mao et al., 2022). On Cityscapes semantic segmentation, robust mIoU improves from $D_N=\{(Z_i,x_i)\}_{i=1}^N$62 to $D_N=\{(Z_i,x_i)\}_{i=1}^N$63 while clean mIoU changes from $D_N=\{(Z_i,x_i)\}_{i=1}^N$64 to $D_N=\{(Z_i,x_i)\}_{i=1}^N$65 (Mao et al., 2022). On PASCAL VOC, robust mAP@50 rises from $D_N=\{(Z_i,x_i)\}_{i=1}^N$66 to $D_N=\{(Z_i,x_i)\}_{i=1}^N$67; on MS-COCO, from $D_N=\{(Z_i,x_i)\}_{i=1}^N$68 to $D_N=\{(Z_i,x_i)\}_{i=1}^N$69 (Mao et al., 2022). Runtime is correspondingly large: per ImageNet image on a single V100 GPU, vanilla inference takes $D_N=\{(Z_i,x_i)\}_{i=1}^N$70 and uses $D_N=\{(Z_i,x_i)\}_{i=1}^N$71, whereas the equivariance method takes $D_N=\{(Z_i,x_i)\}_{i=1}^N$72 and $D_N=\{(Z_i,x_i)\}_{i=1}^N$73 (Mao et al., 2022). The paper therefore frames robustness as an inference-time optimization problem with an explicit clean-accuracy and latency cost.

FoCal pursues a related inference-time goal but through canonicalization by foundation models. Given a transform family $D_N=\{(Z_i,x_i)\}_{i=1}^N$74, it defines

$D_N=\{(Z_i,x_i)\}_{i=1}^N$75

then selects

$D_N=\{(Z_i,x_i)\}_{i=1}^N$76

before running the downstream model on $D_N=\{(Z_i,x_i)\}_{i=1}^N$77 (Singhal et al., 14 Jul 2025). The algorithm proceeds by “Vary and Rank”: enumerate or sample transformations, compute energies using CLIP and a diffusion prior, and use brute force or Bayesian Optimization depending on dimensionality (Singhal et al., 14 Jul 2025). The tested transformation spaces include 2D rotations, 3D viewpoints, illumination shifts, contrast, day–night relighting, and active-vision 6-DoF poses (Singhal et al., 14 Jul 2025).

Quantitatively, FoCal reports several large gains. On Objaverse-LVIS under 3D viewpoint variation, for the worst $D_N=\{(Z_i,x_i)\}_{i=1}^N$78 of input viewpoints, OV-Seg accuracy rises from $D_N=\{(Z_i,x_i)\}_{i=1}^N$79 to $D_N=\{(Z_i,x_i)\}_{i=1}^N$80; overall stability, measured as max–min accuracy, is reduced by $D_N=\{(Z_i,x_i)\}_{i=1}^N$81 (Singhal et al., 14 Jul 2025). On CO3D hard frames, when ground-truth probability is below $D_N=\{(Z_i,x_i)\}_{i=1}^N$82, baseline accuracy $D_N=\{(Z_i,x_i)\}_{i=1}^N$83 rises to $D_N=\{(Z_i,x_i)\}_{i=1}^N$84; at probability below $D_N=\{(Z_i,x_i)\}_{i=1}^N$85, $D_N=\{(Z_i,x_i)\}_{i=1}^N$86 rises to $D_N=\{(Z_i,x_i)\}_{i=1}^N$87 (Singhal et al., 14 Jul 2025). For illumination shifts with CLIP, average gains are $D_N=\{(Z_i,x_i)\}_{i=1}^N$88 percentage points for color and $D_N=\{(Z_i,x_i)\}_{i=1}^N$89 for contrast, with larger gains at extremes (Singhal et al., 14 Jul 2025). On unseen ImageNet rotations, ViT rotated accuracy rises from $D_N=\{(Z_i,x_i)\}_{i=1}^N$90 for PRLC to $D_N=\{(Z_i,x_i)\}_{i=1}^N$91 for FoCal (Singhal et al., 14 Jul 2025). The method also matches PRLC mAP on COCO C4 segmentation while improving pose accuracy from $D_N=\{(Z_i,x_i)\}_{i=1}^N$92 to $D_N=\{(Z_i,x_i)\}_{i=1}^N$93 (Singhal et al., 14 Jul 2025). At the same time, the paper notes the computational cost of $D_N=\{(Z_i,x_i)\}_{i=1}^N$94 energy evaluations, each including CLIP and diffusion passes, and states that runtime remains higher than a single inference (Singhal et al., 14 Jul 2025).

CVP provides a more lightweight inference-time adaptation mechanism. It inserts a tiny convolutional prompt into input space,

$D_N=\{(Z_i,x_i)\}_{i=1}^N$95

where $D_N=\{(Z_i,x_i)\}_{i=1}^N$96 is optimized at test time using a self-supervised contrastive loss (Tsai et al., 2023). With $D_N=\{(Z_i,x_i)\}_{i=1}^N$97 and $D_N=\{(Z_i,x_i)\}_{i=1}^N$98, the prompt has $D_N=\{(Z_i,x_i)\}_{i=1}^N$99 parameters; with $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$00, $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$01 parameters (Tsai et al., 2023). This is described as less than $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$02 of standard visual prompt size (Tsai et al., 2023). On CIFAR-10-C with WideResNet-18, CVP reduces average error from $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$03 to $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$04, a $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$05 point improvement; on ImageNet-C with ResNet-50 it reduces mCE from $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$06 to $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$07 (Tsai et al., 2023). The method also improves CLIP(ViT/32), reducing mCE by $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$08 points and Sketch error by $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$09 points (Tsai et al., 2023). In contrast to heavier inference-time optimization methods, CVP treats robust perception as a small structured prompt-learning problem.

Certification rather than adaptation is the focus of camera motion smoothing. Given any base classifier $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$10 and camera motion distribution $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$11 in 6-DoF motion space, the smoothed classifier is defined as

$y(t):=\phi(Z(t))\approx Cx(t)+v(t)$12

(Hu et al., 2022). The paper develops a certification theorem stating that, if the top-class and runner-up probabilities under smoothing are $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$13 and $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$14, then for a fixed-axis motion $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$15 satisfying

$y(t):=\phi(Z(t))\approx Cx(t)+v(t)$16

the smoothed classifier’s prediction is invariant (Hu et al., 2022). On MetaRoom, motion-smoothed ResNet-18 achieves certified accuracy $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$17 against camera translation along depth within $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$18, $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$19 for horizontal/vertical translation within $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$20, and $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$21 against roll within $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$22 (Hu et al., 2022). Hardware experiments with a Kinova Gen3 arm likewise show consistent gains in empirical robust accuracy across all axes (Hu et al., 2022). This work treats robust perception as a certifiable property under physically meaningful camera perturbations.

6. Temporal, semantic, and task-specific robustness

Not all robustness mechanisms are framed as adversarial defense or uncertainty estimation. Some reconstruct a more stable perceptual representation by exploiting temporal or structural regularities in the input.

“Perception Over Time” introduces a neuro-inspired coarse-to-fine decomposition of a static image into a temporal sequence $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$23 of increasing perceptual clarity (Daniali et al., 2022). The decomposition may be obtained through recurrent sparse coding or approximate JPEG/Gaussian baselines, and the resulting sequence is integrated by a CtF-CNN or CtF-LSTM (Daniali et al., 2022). On ImageNet30, the reported single-frame ResNet baseline achieves standard accuracy $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$24 and adversarial accuracy $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$25 under PGD with $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$26, whereas CtF-CNN with JPEG decomposition reaches $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$27 standard and $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$28 adversarial accuracy, CtF-CNN with RSCD reaches $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$29 and $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$30, and CtF-LSTM with RSCD reaches $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$31 and $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$32 (Daniali et al., 2022). The paper interprets this as robustness emerging from integration of stable low-frequency structure before high-frequency details (Daniali et al., 2022). This suggests a temporal-dynamics view of robust perception even for static image understanding.

RPMArt addresses robustness in articulated-object perception and manipulation under noisy point clouds and sim-to-real transfer (Wang et al., 2024). Its Robust Articulation Network samples $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$33 point tuples, computes translation-invariant tuple features from relative positions, normal-angle features, and learned SHOT embeddings, and predicts joint parameters, affordance-point parameters, and an articulation score (Wang et al., 2024). The articulation-aware classification label is defined so that only tuples straddling the part–base boundary are informative, and tuples with predicted articulation score below $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$34 are discarded at inference (Wang et al., 2024). Under heavy synthetic noise level 4, RoArtNet achieves mean joint-origin error $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$35, direction error $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$36, and affordance-point error $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$37, compared with $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$38, $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$39, and $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$40 for PointNet++ (Wang et al., 2024). In real-world zero-shot transfer, it reports perception errors such as $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$41 origin and $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$42 direction for Microwave, and $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$43 origin and $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$44 direction for WashingMachine (Wang et al., 2024). Here robust perception is explicitly linked to downstream manipulation success, with affordance prediction and joint constraints used to guide actions (Wang et al., 2024).

Robust lane perception has been extended by incorporating traffic flow as a real-time prior. TF-Lane inserts a Traffic Flow-aware Module between the visual backbone and lane decoder, aligning historical tracks into the current ego frame, filtering trajectories by a validity threshold, and fusing traffic-flow features with lane features through block-masked cross-attention (Xie et al., 1 Feb 2026). The work reports that TFM can be plugged into TopoNet, LaneSegNet, MapTR, and MapTRv2 without changing the original task losses or training schedules (Xie et al., 1 Feb 2026). Quantitatively, MapTR on nuScenes improves from $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$45 to $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$46 mAP, LaneSegNet on OpenLaneV2 improves from $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$47 to $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$48 mAP, TopoNet OLS improves from $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$49 to $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$50, and MapTRv2 on nuScenes improves from $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$51 to $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$52 (Xie et al., 1 Feb 2026). A partial-modality inference result is notable: training with TFM but inferring without traffic flow still reaches $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$53 mAP versus $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$54 for the baseline (Xie et al., 1 Feb 2026). The paper interprets this as implicit supervision from the auxiliary modality. A plausible implication is that robust perception may be improved by incorporating structured side information even when that information is absent at test time.

Recent BEV work pushes this idea toward latent world modeling. RESBev reframes robustness as latent semantic prediction at the feature level of Lift-Splat-Shoot pipelines (Zhuo et al., 10 Mar 2026). A latent world model predicts a clean prior feature $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$55 from the previous reconstructed state and ego-motion, and an anomaly reconstructor fuses that prior with the corrupted current BEV feature through cross-attention and a learned per-channel gate (Zhuo et al., 10 Mar 2026). On nuScenes BEV semantic segmentation under RoboBEV corruptions, LSS improves from average IoU $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$56 to $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$57 on seen corruptions and from $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$58 to $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$59 on unseen corruptions when augmented with RESBev; SimpleBEV improves from $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$60 to $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$61 on seen and from $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$62 to $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$63 on unseen corruptions (Zhuo et al., 10 Mar 2026). Under FGSM, PGD, and C&W attacks, vanilla LSS drops below $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$64 IoU while LSS + RESBev recovers above $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$65 IoU (Zhuo et al., 10 Mar 2026). This aligns robust perception with latent state prediction rather than direct denoising.

7. Conceptual themes, trade-offs, and points of tension

The surveyed literature reveals several distinct but interacting meanings of robustness. One is robustness as consensus or trust management, exemplified by ROBOSAC and CoDynTrust, where perception is protected by selecting attacker-free or temporally trustworthy collaborator subsets (Li et al., 2023, Xu et al., 12 Feb 2025). Another is robustness as bounded sensitivity, formalized by the sensitivity metric $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$66 in perception-based control and by certified invariance radii in camera motion smoothing (Makdah et al., 2019, Hu et al., 2022). A third is robustness as uncertainty propagation, where evidential or heteroscedastic predictions are preserved into mapping, localization, or self-calibration layers (Sirohi et al., 2024, Xia et al., 2024). A fourth is robustness as test-time restoration or canonicalization, where inference is modified so that the input or feature representation satisfies structural constraints such as equivariance or typicality (Mao et al., 2022, Singhal et al., 14 Jul 2025). A fifth is robustness as architectural resilience, where the system is explicitly trained for sensor failure, misalignment, or vehicle-specific hardware design (Chen et al., 2023, Dey et al., 2022, Xia et al., 2024).

Several tensions recur across these approaches. The most explicit is the trade-off between accuracy and robustness in perception-based control: lower nominal estimation error forces higher sensitivity to perturbations in the learned noise model (Makdah et al., 2019). A related empirical trade-off appears in inference-time robustness methods, where robust accuracy improves but clean accuracy and runtime can degrade, as in equivariance restoration (Mao et al., 2022). M-BEV, by contrast, reports a small clean-data benefit from robustness-aware training with approximately baseline inference cost in the healthy-sensor case (Chen et al., 2023). This suggests that robustness interventions can operate at different points on the accuracy–efficiency–safety frontier.

Another point of tension concerns whether robustness should be learned at training time or enforced at inference time. ROBOSAC avoids adversarial training and is presented as generalizable to unseen attackers because it relies on output-space consensus rather than attacker-specific priors (Li et al., 2023). Equivariance restoration and FoCal explicitly shift robustness to test-time optimization (Mao et al., 2022, Singhal et al., 14 Jul 2025). PASTA and M-BEV instead redesign or retrain the architecture to survive likely deployment failures (Dey et al., 2022, Chen et al., 2023). This suggests no single consensus in the literature about where robust perception should reside; it may be an attribute of the model, the inference algorithm, or the overall system architecture.

A further tension is between provable guarantees and practical coverage. Robust guarantees for perception-based control and camera motion smoothing derive explicit conditions for safety or certified invariance (Dean et al., 2019, Hu et al., 2022). Yet these guarantees are often conservative; for the quadrotor controller, the theoretical drift bound is approximately $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$67 in simulation and approximately $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$68 on hardware, while empirical worst-case drift is under $y(t):=\phi(Z(t))\approx Cx(t)+v(t)$69 (Jarin-Lipschitz et al., 2020). By contrast, high-performing empirical methods such as FoCal, RESBev, or CoDynTrust report large gains under varied corruptions but do not provide formal guarantees (Singhal et al., 14 Jul 2025, Zhuo et al., 10 Mar 2026, Xu et al., 12 Feb 2025). A plausible implication is that robust perception research remains split between certifiable but specialized methods and broader empirical defenses with weaker formal assurances.

Finally, many works converge on the idea that robust perception is inseparable from the structure of the downstream task. In control, robustness is meaningful only relative to closed-loop safety and invariance (Makdah et al., 2019, Dean et al., 2019, Jarin-Lipschitz et al., 2020). In collaborative perception, uncertainty and trust are valuable partly because they can be propagated to planning and control (Xu et al., 12 Feb 2025). In localization and mapping, uncertainty is only useful if it modifies particle weights or map quality (Sirohi et al., 2024). In articulated-object manipulation, perception robustness is evaluated through action success under joint constraints (Wang et al., 2024). This suggests that “robust perception” is not merely about making a predictor less fragile in isolation. It is about preserving operational validity under perturbation in the context of a larger embodied, multi-agent, or decision-theoretic system.