Pseudo-task Substitution Training (PST)

• The paper introduces PST as a replay-free mechanism that synthesizes pseudo-features with moment matching to bridge isolated routers in continual adversarial training.
• PST leverages running mean and covariance to statistically reconstruct feature distributions, eliminating the need for raw data storage.
• Empirical results show that integrating PST boosts adversarial robustness by 2–3 percentage points while reducing memory overhead compared to traditional rehearsal methods.

Searching arXiv for the specified papers to ground the article in current preprint records. Pseudo-task Substitution Training (PST) is a feature-space replay strategy introduced in the continual adversarial training framework "Dynamic Dual-level Defense Routing for Continual Adversarial Training" (Wang et al., 24 Sep 2025). It addresses a specific failure mode of continual adversarial training (CAT): as routers $$\{\mathcal{R}^0,\mathcal{R}^1,\dots,\mathcal{R}^t\}$$ are incrementally added and each is trained only on its own stage’s adversarial data, the routers remain isolated from one another, and the Adversarial Sentinel Network (ASN) cannot learn a unified routing criterion for test-time inputs whose true attack type is unknown. PST replaces raw-data rehearsal with pseudo-features synthesized from stored feature statistics, allowing ASN to train over a mixed multi-stage distribution without storing historical images. Within Dual-level Defense Routing (DDeR), this mechanism supports second-level routing by making past attack stages visible to ASN while preserving the first-level arrangement of routers and defense experts (Wang et al., 24 Sep 2025).

1. Problem setting in continual adversarial training

The motivating setting is CAT under evolving adversarial attacks. Existing CAT approaches typically balance decision boundaries by either data replay or optimization strategy to constrain shared model parameters, but these methods suffer from catastrophic forgetting of previous defense knowledge after continual learning because adversarial examples are diverse and aggressive (Wang et al., 24 Sep 2025).

In DDeR, the first-level defense routing comprises multiple defense experts and routers, with each router dynamically selecting and combining suitable experts to process attacked features. Routers are independently incremented as continuous adversarial training progresses, and their selections are guided by ASN in the second-level defense routing. The independence of routers creates the core difficulty that PST is designed to resolve: without a mechanism that exposes ASN to representations of past stages, ASN cannot learn to route incoming examples to the correct router at test time.

Traditional rehearsal would solve this by replaying historical data, but the formulation explicitly identifies large storage and privacy concerns. PST therefore creates feature-level “pseudo-tasks” drawn from approximated past distributions. This suggests that PST is not primarily a defense expert-learning mechanism; it is an inter-router communication mechanism for routing supervision under replay-free continual learning.

2. Statistical construction of pseudo-tasks

At stage $t$, the method denotes by $\mathbf{f}_t \in \mathbb{R}^d$ the $d$-dimensional feature vector produced by the shared CLIP image encoder for an example drawn from attack type $t$ (Wang et al., 24 Sep 2025). For each stage, PST maintains a running mean vector and covariance matrix,

$$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$

If $L_t$ is the lower-triangular Cholesky factor of $\Sigma_t$, then $\Sigma_t = L_t L_t^\top$.

During training, the paper states that the instantaneous batch covariance at stage $t$ is

$t$0

after which the stored $t$1 is updated via a moving average, as in standard momentum-based feature statistics. The exact momentum hyperparameter is not specified in the formulation, although implementation guidelines later state that momentum for covariance was set comparably to batch-norm ($t$2).

Pseudo-features for a past attack $t$3 are sampled by reparameterization:

$t$4

This produces synthetic feature vectors whose first- and second-order moments match the stored $t$5. ASN is then trained to predict the correct router index $t$6 from a real or pseudo feature. Denoting ASN’s logits by $t$7, the training loss is

$t$8

where $t$9 is the softmax over routers and $\mathbf{f}_t \in \mathbb{R}^d$0 is the mixture of real features when $\mathbf{f}_t \in \mathbb{R}^d$1 and pseudo-features when $\mathbf{f}_t \in \mathbb{R}^d$2.

A notable property of the formulation is the absence of an explicit divergence term. Unlike methods that explicitly minimize a KL- or Wasserstein-distance between past and current distributions, PST enforces distributional alignment implicitly through moment-matched feature-space sampling.

3. Algorithmic role inside dual-level defense routing

PST is integrated into DDeR as a stage-wise training procedure for routing rather than as a modification of the defense experts themselves (Wang et al., 24 Sep 2025). At each stage $\mathbf{f}_t \in \mathbb{R}^d$3, the framework first trains or expands router $\mathbf{f}_t \in \mathbb{R}^d$4 and its experts on the adversarial dataset $\mathbf{f}_t \in \mathbb{R}^d$5 via DRDE+DEU. It then computes a feature batch $\mathbf{f}_t \in \mathbb{R}^d$6 using the frozen CLIP image encoder $\mathbf{f}_t \in \mathbb{R}^d$7.

From these features, PST updates the current stage statistics by moving averages of the mean and covariance. It next constructs a mixed-feature set $\mathbf{f}_t \in \mathbb{R}^d$8 containing the current stage’s real features labeled with stage index $\mathbf{f}_t \in \mathbb{R}^d$9, together with pseudo-features sampled for each earlier stage $d$0 using

$d$1

and labeling those pseudo-features with $d$2.

ASN is trained on minibatches drawn from this mixed set. The paper specifies that backpropagation is applied with respect to ASN’s context vectors only. After training, pseudo-features are discarded, and only $d$3 for $d$4 are retained. The inputs to PST at stage $d$5 are therefore the current stage’s real features and the stored statistics of previous stages; the outputs are updated ASN context vectors and a routing head capable of assigning any incoming $d$6 to one of $d$7.

PST also has explicit interaction with ASN’s cross-modal (image–text) prompt learner. At test time, ASN is run on $d$8 and triggers router $d$9. This makes PST operationally distinct from rehearsal-based CAT: the method does not preserve historical images, but it does preserve stage-indexed feature statistics that support routing over all learned stages.

4. Assumptions, limits, and theoretical status

The paper does not provide explicit theorems or bounds on forgetting or convergence for PST (Wang et al., 24 Sep 2025). Its theoretical justification is stated as intuition rather than formal analysis: by capturing only first- and second-order moments, PST affords a lightweight surrogate for full replay with provably matching means and covariances, and empirically this suffices to bind the routers’ decision regions together.

That design implies a strong representational assumption. PST summarizes each past stage only through $t$0 and reconstructs pseudo-features by Gaussian-style sampling through the Cholesky factor. A plausible implication is that the method is optimized for retaining coarse distributional structure rather than exact historical feature manifolds. The formulation does not claim recovery of higher-order moments, nor does it introduce an explicit discrepancy objective beyond the moment matching induced by sampling.

The implementation details reinforce this lightweight orientation. The hyperparameters introduced by PST are the number of pseudo-samples per past stage, denoted $t$1 in practice, the momentum coefficients for running mean and covariance updates, and the relative weighting between real and pseudo examples in ASN minibatches. The reported guideline is that a moderate number, such as $t$2–$t$3 synthetic features per past stage, was sufficient in the authors’ experiments, with covariance momentum set comparably to batch-norm ($t$4).

5. Empirical behavior and memory profile

The reported ablation study isolates PST’s contribution within the DDeR pipeline (Wang et al., 24 Sep 2025). The extracted Table 5 results are as follows.

Method	Accuracies (FGSM, PGD, CW, AA, Df, Clean)	Note
(a) Baseline	48.22, 49.73, 46.46, 50.38, 58.13, 49.33	Reference
(b) + DRDE	54.17, 55.93, 55.26, 56.39, 62.32, 54.46	Router/expert addition
(c) + DRDE + PST (w/o ASN)	56.67, 57.36, 56.43, 57.22, 63.79, 56.17	PST without ASN
(d) + DRDE + ASN (w/o PST)	63.91, 64.63, 65.68, 65.47, 67.73, 66.46	ASN without PST
(e) + DRDE + ASN + PST (Ours)	68.56, 67.38, 68.53, 68.49, 69.79, 70.34	Full method

The paper states that introducing PST on top of DRDE, comparing (c) with (b), yields a gain of approximately $t$5–$t$6 points across attacks, confirming that pseudo-feature replay mitigates forgetting in ASN. Full PST+ASN integration, row (e), produces the largest overall boost.

The memory argument is equally central. Compared to rehearsal-based CAT such as SSEAT, PST requires only the storage of $t$7 for each stage, approximately $t$8 floats, rather than full raw images. In ImageNet-1K experiments, DDeR’s peak memory was reported as $t$9 versus $$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$0 for SSEAT. In the line-plot of continual accuracy across the attack sequence [FGSM, PGD, CW, AA, Df], DDeR with PST retains approximately $$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$1 accuracy on past attacks even after five stages, whereas a pure replay method drops below $$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$2.

These results support the specific claim made for PST in the paper: it is a lightweight, feature-space replay strategy that overcomes the independent-router isolation problem in CAT by synthesizing a multi-stage distribution for ASN training.

6. Terminological scope and disambiguation

The acronym “PST” is not unique across arXiv, and a separate 2024 paper uses it to denote “Pedestrian-Sensitive Training,” a training-time add-on for two-stage CNN pedestrian detectors rather than a continual adversarial training strategy (Guo, 2024). In that work, PST addresses false positives generated by rigid IoU-based proposal labeling. The method inserts a lightweight pedestrian-sensitive classifier $$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$3 during training only, re-inspects proposals in the IoU-low set $$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$4, and rejects those whose pedestrian score exceeds a threshold $$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$5 such as $$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$6. The refined negative set is

$$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$7

and the second-stage detector is trained on

$$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$8

That detector-oriented PST keeps the RPN/backbone and second-stage classifier/regressor unchanged, adds a 9-layer CNN used only at training time, and preserves inference speed and model size. Its reported metrics are in miss rate rather than adversarial robustness: on CityPersons, VGG16 + Faster R-CNN is listed at approximately $$\boldsymbol{\mu}_t = \mathbb{E}[\mathbf{f}_t], \qquad \Sigma_t = \mathbb{E}\bigl[(\mathbf{f}_t - \boldsymbol{\mu}_t)(\mathbf{f}_t - \boldsymbol{\mu}_t)^\top\bigr] \in \mathbb{R}^{d \times d}.$$9 MR and improves to $L_t$0 MR with PST; on the SY-Metro dataset, MR drops from $L_t$1 to $L_t$2; MetroNext-PST retains a $L_t$3 parameter footprint and reports $L_t$4 on Jetson Nano CPU.

This terminological overlap matters because the two methods solve different problems with different training signals. In (Wang et al., 24 Sep 2025), PST is a replay-free, feature-statistics-based mechanism for ASN supervision in continual adversarial training. In (Guo, 2024), PST is a proposal-filtering strategy for suppressing false positives in two-stage pedestrian detection. The shared acronym should therefore not be treated as evidence of a shared algorithmic lineage.

7. Significance within DDeR

Within DDeR, PST is best understood as the component that makes second-level defense routing trainable under continual attack shifts (Wang et al., 24 Sep 2025). The first-level routing mechanism can expand with new routers and experts, but without PST the independence of routers prevents ASN from learning a single routing head over historical and current attack stages. PST closes that gap by summarizing each past stage through feature mean and covariance, sampling pseudo-features with matched first- and second-order moments, and using those samples to supervise ASN across all stages.

The formulation also clarifies several common misconceptions. PST is not raw-image replay; it stores only $L_t$5. It is not an explicit distribution-matching objective based on KL or Wasserstein distance; alignment is implicit through moment-matched sampling. It is not accompanied by formal convergence guarantees in the paper. Its contribution is operational: it enables inter-router communication without storing historical data, improves robust accuracy by approximately $L_t$6–$L_t$7 percentage points when added on top of DRDE alone, and, when combined with ASN in the full DDeR system, yields the strongest reported continual defense performance and classification accuracy in the study.