RAMS: Residual-Based Adversarial Sampling

• The paper introduces RAMS, a gradient-based adaptive sampling method that maximizes PDE residuals to enhance neural solver accuracy.
• It applies projected gradient ascent to reposition collocation points, efficiently targeting high-error regions in both physics-informed and data-driven settings.
• Empirical results show significant error reductions and minimal computational overhead, making RAMS effective for high-dimensional PDE and operator learning tasks.

Residual-Based Adversarial-Gradient Moving Sample (RAMS) is a gradient-based adaptive sampling methodology designed for scientific machine learning, particularly for improving the efficiency and accuracy of neural network solvers for partial differential equations (PDEs). RAMS adaptively relocates collocation points or operator evaluation samples toward high-error (high-residual) regions by performing gradient ascent on the PDE residual, making it computationally tractable for high-dimensional problems and operator learning settings (Ouyang et al., 1 Sep 2025).

1. Mathematical Foundations

RAMS operates within the paradigm of physics-informed neural networks (PINNs) and neural operator models for solving or learning PDEs. In traditional PINN setups, the network $N_\theta:\Omega\to\mathbb R$ is trained on a fixed or static collocation set $\{x_i\}$, minimizing the mean squared residual loss

$$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$

where $r(x)$ denotes the pointwise PDE residual, e.g., $r(x) = \mathcal F[N_\theta](x)$. RAMS approaches the selection (and movement) of sampling points as an adversarial optimization problem, aiming to maximize $r(x)^2$ over $x \in \Omega$ for each sample, or, in the full minimax form,

$$\min_{\theta}\;\max_{\{x_i\}\subset\Omega} \frac{1}{N} \sum_{i=1}^N (r(x_i))^2.$$

The adversarial gradient for adjusting a sample is given by

$g(x) = \nabla_x(r(x)^2) = 2\,r(x)\,\nabla_x r(x).$

Updating points follows a projected gradient ascent: $$x_{\rm new} = \Pi_\Omega\left[x + \alpha\,g(x)\right],$$ where $\{x_i\}$0 is a step size and $\{x_i\}$1 denotes projection back onto the feasible domain.

2. Algorithmic Procedure

RAMS integrates directly into the training workflow of PINNs or operator networks by alternating between neural network parameter updates and sample relocations. A canonical RAMS update on a batch of $\{x_i\}$2 trainable points consists of:

• Computing the residual $\{x_i\}$3 and its gradient for each $\{x_i\}$4.
• Applying a vectorized gradient ascent step to shift $\{x_i\}$5 in the direction of increasing residual.
• Projecting updated points back into the admissible domain.

This process interleaves with standard network optimization: after each set number of parameter updates, RAMS moves a fraction of collocation points, then resumes network training. For operator learning, RAMS generalizes by moving function-parametric inputs rather than coordinates, enabling application to both physics-informed and data-driven paradigms (Ouyang et al., 1 Sep 2025).

Pseudocode Overview

$r(x)$9 Integration into PINN or operator learning entails alternately optimizing network parameters over fixed and moved samples, with RAMS steps applied at prescribed intervals.

3. Computational Efficiency and Theoretical Properties

The per-iteration complexity of RAMS is $\{x_i\}$6, with $\{x_i\}$7 the number of trainable samples and $\{x_i\}$8 the cost of a residual and gradient evaluation. Compared to residual-based refinement (e.g., RAR), which requires scanning and sorting over large candidate pools (costing $\{x_i\}$9 for $$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$0), RAMS achieves order-of-magnitude gains in high dimensions since it avoids such exhaustive search and directly relocates samples (Ouyang et al., 1 Sep 2025). Empirical overhead is reported as less than 3% of training time for benchmark problems.

Convergence of the sample-moving dynamics depends on the smoothness of $$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$1 and selection of the step size $$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$2. Properly tuned, sample locations converge toward high-residual regions; excessive $$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$3 may destabilize updates. In high-dimensional spaces, smaller gradients necessitate $$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$4 scaling. Empirical results suggest that moving a fraction of points per stage (10–30%) stabilizes network training.

4. Experimental Validation and Performance

RAMS demonstrates robust empirical improvements across a suite of scientific benchmarks:

• PINN examples: 1D Burgers, 2D and 10D Poisson, 1D Wave equations.
• Operator learning: Physics-informed and data-driven settings, including multidimensional and high-dimensional (up to $$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$5) tasks.

Error reductions (mean relative $$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$6 error) are summarized as follows:

Sampler	Burgers w/o RAMS	Burgers w/ RAMS	2D Poisson w/o RAMS	2D Poisson w/ RAMS	10D Poisson w/o RAMS	10D Poisson w/ RAMS
Random	0.181	0.010	0.285	0.089	0.98	0.006
RAR-G	0.045	0.010	0.074	0.018	0.52	0.004
R3	0.060	0.012	0.093	0.025	0.75	0.007

On the 10D Poisson equation, Random+RAMS achieves errors $$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$7 with $$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$8 points, while random alone yields $$L_{\rm phy}(\theta) = \frac{1}{N}\sum_{i=1}^N (r(x_i))^2,$$9. Operator learning tasks exhibit error reductions of 30–70% using RAMS, matching the accuracy of more expensive flow-based samplers using only a small fraction of their sampling budget (Ouyang et al., 1 Sep 2025).

5. Integration and Extension to Operator Learning

RAMS is the first efficient adaptive sampling method for operator learning, including both physics-informed and data-driven operator scenarios. In PI operator learning, each sample is a tuple (e.g., $r(x)$0), and RAMS moves only the function-parameter component via kernel smoothing. In data-driven active learning, RAMS selects and moves a subset of candidate functions, solves the corresponding PDEs for ground truth, and augments the training set.

This unified gradient-based framework allows seamless integration with arbitrary baseline samplers (random, LHS, RAR, R3, importance sampling). RAMS’ relocation mechanism is general, applicable in high-dimensional input spaces and with diverse PDE types (Ouyang et al., 1 Sep 2025).

6. Practical Implementation Considerations

Key hyperparameters for RAMS include the sample movement step size $r(x)$1, the number of inner RAMS steps $r(x)$2, and the frequency of resampling. Recommendations are:

• $r(x)$3, set so average displacement is $r(x)$4 of the domain per step.
• $r(x)$5 for low dimension, $r(x)$6 for high-dimension.
• Resample every $r(x)$7 network updates.

Limitations include the cost of calculating $r(x)$8 for complex or discontinuous PDEs (where smoothing or alternative techniques may be needed) and the local nature of the update rule. Potential extensions include adaptive line search, trust-region methods, or second-order Newton-type sample moves for improved robustness (Ouyang et al., 1 Sep 2025).

7. Significance and Outlook

By employing adversarial-gradient relocation of samples, RAMS adaptively targets regions where networks underperform, efficiently allocating computational resources for training PINNs and neural operators. This facilitates order-of-magnitude accuracy improvements in both physics-informed and data-driven learning without inflating overall sample counts or imposing prohibitive runtime overhead. RAMS represents a unified and extensible paradigm for adaptive sampling, advancing the practical solvability of high-dimensional and operator learning PDE problems (Ouyang et al., 1 Sep 2025).