- The paper shows that randomly deleting 10–50% of stored trajectories per training round can reduce adaptive RL robustness gaps by up to 30% for MLP estimators and 6% on average for recurrent models.
- The method applies exponential recency weighting while preserving data diversity, allowing a Narrow MLP with deletion to outperform a Wide MLP without deletion despite using roughly five times fewer parameters.
- The theoretical analysis finds deletion can improve test risk under train–deployment distribution shift, but benefits depend on model architecture, retention rate, environment, and the severity of mismatch.
Overview
This paper studies adaptive reinforcement learning in the contextual Markov Decision Process (cMDP) framework, where a family of environments M(c)=(S,A,Pc,R) is indexed by a low-dimensional context c that is unknown at test time. The standard decomposition—train a universal policy K:C→Π assuming access to the true context, then pair it with a context estimator ϕ that infers c from the observed state-action trajectory—inherits a fundamental distribution mismatch: the estimator is trained on trajectories collected by earlier policies but must perform on trajectories generated by its own current policy. The paper's central contribution is a simple intervention for this mismatch: after each training round, delete a randomly chosen fraction 1−α of the stored trajectories. Because data accumulates across rounds under progressively better policies, random deletion induces an implicit exponential decay on older samples without requiring any explicit identification of which samples are stale. The authors report that this reduces the robustness gap by up to 30% for MLP estimators and 6% on average for recurrent networks, and they support the empirical finding with a theoretical analysis of regularized ERM under train/test distribution shift.
Method
The training procedure follows Yu et al.'s universal policy with online system identification. The universal policy is trained on augmented states zt=[st,c] using any off-the-shelf RL algorithm (DDPG for Pendulum, PPO for LunarLander, etc.), with contexts resampled from the training set each episode. The context estimator ϕθ, taking the last k=4 state-action pairs (or a recurrent hidden state summarizing longer history), is trained by supervised regression to minimize ∥ϕθ(sτ:τ−k,aτ−1:τ−k)−c∥2. Multi-round training is essential: trajectories collected only under c0 never exhibit context-policy mismatch (c1), which the estimator inevitably encounters at deployment.
The proposed modification is minimal: at the end of each round, permanently delete a c2 fraction of buffer trajectories uniformly at random. With retention fraction c3, a trajectory's survival probability decays as c4 across rounds—an exponential down-weighting of stale data. The mechanism is a compromise between two extremes: keeping all data preserves diversity and coverage of the state-action-context space but retains many samples from outdated policies; discarding all old data removes exactly those samples whose distribution no longer matches deployment conditions but sacrifices diversity. Random deletion approximates the desired recency weighting while retaining some coverage from every round.
Empirical results
Evaluation uses the robustness gap relative to UP-true (the universal policy given the true context), defined as c5, across Gym classical control/Box2D environments (Pendulum, LunarLander, Acrobot, MountainCar) and Brax locomotion tasks (Ant, Halfcheetah), with contexts spanning gravity, link mass, link length, friction, and torso mass. Four estimator architectures are compared: a Wide MLP (47,426 parameters), a Narrow MLP (8,322), an LSTM (7,362), and a GRU (5,538).
Three findings stand out:
| Architecture |
Best c6 |
No deletion (c7) |
| Wide MLP |
0.62 |
0.57 |
| Narrow MLP |
0.53 |
0.82 |
| LSTM |
0.43 |
0.50 |
| GRU |
0.43 |
0.48 |
First, deletion helps small models but slightly hurts the Wide MLP. For the Narrow MLP, LSTM, and GRU, deleting 10–50% of data per round improves average robustness gap; for the Wide MLP, full retention performs best. Second, deletion enables capacity reduction: the Narrow MLP with deletion (gap 0.53) outperforms the Wide MLP without deletion (gap 0.57), despite having roughly 5× fewer parameters—a result with direct implications for inference cost on embedded platforms. Third, recurrent architectures dominate overall: LSTM and GRU achieve average gaps of 0.42 versus 0.55 and 0.52 for the wide and narrow MLPs, and in the most extreme case (LunarLander, GRAVITY_X), the LSTM's gap of 0.24 versus the Narrow MLP's 0.93 represents a 69-percentage-point improvement.
An ablation against alternative deletion strategies clarifies the mechanism. Stale deletion (removing only the oldest trajectories) is not a clear winner for any architecture, indicating that older trajectories still contribute useful diversity rather than being purely out-of-distribution. Uniform deletion (stochastic subsampling without permanent removal) is best for the Narrow MLP, while random deletion is best for the recurrent models. This heterogeneity suggests the optimal staleness/diversity trade-off is architecture-dependent, and the paper does not offer a predictive rule for choosing among deletion strategies a priori.
The retention fraction itself requires per-environment tuning over c8; c9 is optimal in 4 of 6 environments for the LSTM, but Pendulum prefers K:C→Π0. This tuning burden is a practical caveat: the reported gains assume access to evaluation feedback for hyperparameter selection.
Theory: beneficial deletion under distribution shift
To explain when deletion helps, the paper analyzes regularized ERM where training data is drawn i.i.d. from K:C→Π1 but test loss is evaluated under a different distribution K:C→Π2. Under smoothness of loss and regularizer plus strong convexity of K:C→Π3, the main theorem gives a sufficient condition for deleting a single uniformly random point to strictly decrease expected test loss:
K:C→Π4
where K:C→Π5 captures whether the parameter shift induced by deletion is a descent direction for the test risk, and K:C→Π6 bounds curvature along the deletion path via a homotopy argument on K:C→Π7. The proof Taylor-expands the expected test loss along this path, showing the first-order term dominates when the condition holds.
Two structural observations give the result its meaning. First, the condition cannot hold without distribution shift: if K:C→Π8, then as K:C→Π9 the empirical gradient average converges to ϕ0, forcing ϕ1. Deletion is thus provably tied to mismatch, not a generic regularization effect. Second, specializing to ridge regression with noisy training labels (ϕ2) and clean test labels yields explicit sufficient conditions: deletion helps when the regularization coefficient lies in a moderate range and the signal-to-noise ratio satisfies ϕ3 for constants meeting a positivity constraint (e.g., ϕ4, ϕ5 gives threshold 0.44). Crucially, the SNR bound lower-bounds ϕ6—the noise-to-signal ratio—which directly quantifies how large the train/deployment mismatch must be for deletion to be beneficial.
Supporting supervised experiments extend the phenomenon beyond RL: on MNIST (train digits {0,1,2}, test {0,1}) and CIFAR (train {airplane, automobile, bird}, test {airplane, automobile}), excluding 5% of training data improves test accuracy for at least one setting of dropout or weight decay, confirming that beneficial deletion arises in ordinary classification with contaminated training distributions.
Several caveats limit the theory's direct applicability to the RL experiments. The theorem covers deletion of a single point, whereas practice deletes 10–50% of the buffer; the analysis treats ϕ7 and ϕ8 as fixed and independent, whereas in adaptive RL both depend on the sequence of previous estimators ϕ9; and strong convexity of c0 does not hold for neural network objectives. The theoretical results should therefore be read as an idealized explanation of the mechanism rather than a guarantee for the experimental setting.
Limitations and open questions
The paper concedes several limitations explicitly. The Wide MLP does not benefit from deletion, and the paper offers no criterion predicting for which architectures or capacities deletion will help. The retention fraction c1 must be tuned per environment-architecture pair, and the comparison among random, stale, and uniform deletion shows architecture-dependent winners without a principled selection rule. On the theoretical side, the gap between the single-point, fixed-distribution, strongly convex analysis and the multi-round, self-referential, nonconvex RL setting remains open, as does extending the ridge-regression SNR characterization to general losses or to the sequential dependence of c2 and c3 on prior estimators.
Conclusion
This paper identifies random data deletion as a low-cost intervention that improves context estimation in adaptive RL by implicitly expiring stale trajectories while preserving diversity. Empirically, it reduces robustness gaps for compact estimators—up to 30% for MLPs—and allows a 5× smaller MLP to surpass a wider one trained without deletion. Theoretically, within regularized ERM under distribution shift, uniform deletion provably reduces expected test loss when the induced parameter change aligns with the test-risk descent direction, with the ridge-regression case yielding an explicit SNR threshold that quantifies the required mismatch. The work leaves open a predictive characterization of when deletion helps as a function of model capacity and the coupled, history-dependent distributions arising in multi-round adaptive training.