Prior-Guided Data Distillation (PGD)

• PGD is a method that uses domain-specific priors to constrain the selection of high-value training samples, ensuring robust and representative distillation.
• It employs a teacher-student framework in diverse settings—such as neural operator learning for PDEs, offline reinforcement learning, and dataset distillation—to improve model generalization.
• The approach focuses on optimizing hard regions in the input space through selective sampling, reducing computational overhead while boosting efficiency and performance.

Searching arXiv for the specified papers and related uses of “Prior-Guided Data Distillation (PGD)”. Prior-Guided Data Distillation (PGD) denotes a family of distillation procedures in which prior information constrains the synthesis, selection, or reweighting of training inputs, and the resulting informative samples are distilled from a stronger source into a compact model, planner, or synthetic dataset. In recent arXiv usage, the term is not fully standardized. In operator learning, it refers to PGD-style active sampling in function space under smoothness and energy constraints, with a differentiable numerical PDE solver acting as teacher (Sun, 21 Oct 2025). In offline reinforcement learning, it refers to learning a behavior-regularized latent prior for a frozen diffusion planner so that high-value trajectories are generated directly from a refined initial distribution (2505.10881). In dataset distillation, closely related prior-guided formulations use frozen diffusion backbones as priors and inject a representativeness term into the reverse diffusion dynamics without retraining (Su et al., 20 Oct 2025). A distinct but related acronym, Prediction-Guided Distillation, appears in dense object detection, where teacher predictions determine which spatial regions should be distilled most strongly (Yang et al., 2022).

1. Conceptual structure and terminology

Across these formulations, a common pattern is the use of a prior to restrict or bias the search space for informative examples, followed by distillation from a stronger reference model. In the PDE setting, the prior is an admissible set of perturbations

$$a^\star = \arg\max_{\|a'-a\|_2 \le \epsilon,\ a'\in\mathcal{C}} \ell(a';\theta),$$

where $\mathcal{C}$ encodes norm bounds, feasible value ranges, and implicit smoothness or periodicity constraints (Sun, 21 Oct 2025). In offline RL, the prior is a learnable latent distribution $p_\psi(\mathbf{x}_T|s)$ trained with a behavior-regularized objective

$$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$

with the denoiser $g_s$ kept fixed after behavior cloning (2505.10881). In dataset distillation, prior guidance appears as a conditional score decomposition

$$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$

where the second term is a representativeness prior defined by kernel similarity in diffusion feature space (Su et al., 20 Oct 2025).

This suggests that PGD is best understood as a methodological motif rather than a single algorithm. The motif has three recurring components: a compact student or sampler, a stronger teacher or prior-bearing model, and an explicit mechanism for concentrating supervision on hard, high-value, or representative regions of the input space. The literature also shows that the acronym is overloaded: in dense detection, PGD denotes Prediction-Guided Distillation rather than Prior-Guided Data Distillation, even though the method is also prior-driven in a broader sense (Yang et al., 2022).

2. PGD in neural operator learning for PDEs

In operator learning, PGD is motivated by the limitations of both classical PDE solvers and compact neural surrogates. Standard nonlinear PDE solvers such as finite difference, finite volume, finite element, and spectral methods rely on very fine spatial grids, small time steps, local linearizations or Taylor expansions, and stability constraints such as CFL conditions. For a time-dependent PDE

$u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$

a traditional solver iteratively applies

$$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$

so that $u^N \approx u(\cdot,T)$ with $N=T/\Delta t$ potentially very large. In the spectral-solver formulation used as teacher,

$\mathcal{C}$0

These properties produce large memory cost, long wall-clock time, and expensive repeated solves when many initial conditions are required (Sun, 21 Oct 2025).

Neural operators such as FNOs and DeepONets instead learn the solution operator

$\mathcal{C}$1

with the student written generically as $\mathcal{C}$2. FNOs implement global kernels through truncated Fourier transformations, while DeepONet combines a branch net for the input function and a trunk net for query locations. These models provide fast single-shot inference and low parameter cost, but the paper documents poor OOD behavior. In Burgers, an FNO trained on initial velocities in range $\mathcal{C}$3 predicts wrong wave propagation speed under larger magnitudes such as $\mathcal{C}$4 and even wrong propagation direction under sign-flipped inputs such as $\mathcal{C}$5. In Navier–Stokes, errors blow up when Gaussian random field kernels or ranges differ from training, with outputs biased toward extremes. The paper attributes these failures to spectral truncation and low-pass bias, data-driven interpolation on a narrow training manifold, and the sensitivity of nonlinear PDEs to initial conditions (Sun, 21 Oct 2025).

The teacher–student formulation uses a differentiable spectral PDE solver in JAX as teacher,

$\mathcal{C}$6

and a compact neural operator $\mathcal{C}$7 as student. The distillation objective is

$\mathcal{C}$8

For Burgers, the task loss is mean squared error,

$\mathcal{C}$9

For Navier–Stokes, the study additionally considers pixel-wise MSE, soft-DTW-like losses, and perceptual losses via VGG, motivated by periodic ambiguities and rigid translations (Sun, 21 Oct 2025).

The central PGD step is a PGD-style active sampling loop in function space. Starting from seed inputs drawn from a base distribution, the method locally searches for perturbations that maximize student–teacher discrepancy while respecting physically meaningful priors:

• discrete $p_\psi(\mathbf{x}_T|s)$0 norm bounds, $p_\psi(\mathbf{x}_T|s)$1;
• value-range clipping, $p_\psi(\mathbf{x}_T|s)$2;
• implicit smoothness and periodicity induced by spectral discretization.

With discretized functions represented as tensors $p_\psi(\mathbf{x}_T|s)$3, the $p_\psi(\mathbf{x}_T|s)$4 PGD update uses normalized gradients and projection back onto the $p_\psi(\mathbf{x}_T|s)$5 ball, with an Adam-like adaptive PGD variant also implemented. The paper distinguishes three gradient constructions for the inner problem: full gradients through both student and solver, detached-solver gradients, and a nearest-neighbor dictionary approximation in the style of Adesoji et al. Full gradients are reported to yield stronger, more stable attacks and higher loss increases; detached gradients are weaker; dictionary-based approximations can be misleading (Sun, 21 Oct 2025).

The outer loop is an active learning or data distillation loop. The student is trained on the current dataset, PGD is run on existing training inputs to mine hard adversarial inputs, the solver labels these inputs, and the training set is then augmented or partially replaced. The paper also studies batch-by-batch adversarial training, but finds it less effective for Navier–Stokes because the solver is extremely memory-hungry and small batch sizes degrade in-distribution performance. Round-by-round active distillation yields improved OOD performance on many distributions, particularly those with the same range but different kernels or parameters, while preserving the fast inference and compactness of neural operators. At the same time, the method does not make FNOs “universal solvers,” and certain extreme ranges remain difficult (Sun, 21 Oct 2025).

3. Latent-prior PGD for offline diffusion planning

In offline reinforcement learning, PGD appears in a different form. The setting is a standard MDP with a fixed dataset $p_\psi(\mathbf{x}_T|s)$6 generated by a behavior policy $p_\psi(\mathbf{x}_T|s)$7, and the central concern is distributional shift: actions far from those represented in $p_\psi(\mathbf{x}_T|s)$8 can cause critic over-optimism and unstable policy improvement. Diffusion planners address long-horizon planning by generating trajectories through iterative denoising, but existing guidance strategies have specific drawbacks. Classifier Guidance can collapse multimodal distributions; Classifier-Free Guidance can drift under conditioning outside the training range; and Monte Carlo Sample Selection incurs high inference cost and no explicit behavior regularization (2505.10881).

The proposed remedy is to replace the standard Gaussian prior of a behavior-cloned diffusion planner with a learnable prior distribution, while leaving the denoiser fixed. In the underlying DDPM-style model,

$p_\psi(\mathbf{x}_T|s)$9

and in the planner formulation a deterministic DDIM operator $$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$0 maps $$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$1 to a trajectory $$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$2:

$$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$3

Prior Guidance introduces a learnable latent prior

$$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$4

typically parameterized as a conditional Gaussian or Gaussian mixture with mean and log-standard deviation predicted by a GRU. The induced trajectory distribution becomes

$$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$5

The paper states that it is natural to refer to this prior-learning process as Prior-Guided Data Distillation because the prior distills high-return structure from the offline data and critic into a compact generative prior over diffusion latent space (2505.10881).

A key theoretical step is a latent-space reformulation of behavior regularization. Under an approximate bijectivity assumption for $$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$6 with sufficiently fine DDIM steps, the density ratio in trajectory space simplifies to the density ratio in latent space:

$$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$7

This converts an intractable trajectory-space problem into

$$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$8

where the regularizer is now a divergence between Gaussians and therefore admits closed form for KL, reverse KL, and Pearson $$\max_\psi \mathbb{E}_{s,\mathbf{x}_T\sim p_\psi(\cdot|s)} \left[ V\big(g_s(\mathbf{x}_T)\big) - \alpha f\!\left(\frac{p_\psi(\mathbf{x}_T|s)}{p(\mathbf{x}_T)}\right) \right],$$9 (2505.10881).

To avoid gradients through the full denoising process, the method introduces a latent critic $g_s$0 trained by regression to approximate $g_s$1. The prior is then optimized using

$g_s$2

This removes backpropagation through the denoiser entirely. At test time, the planner samples a single latent $g_s$3, denoises it to a trajectory with $g_s$4, and uses an inverse dynamics model to extract the action. Relative to MCSS, the resulting controller does not require inference-time multi-sample optimization or repeated critic calls (2505.10881).

Empirically, the method is evaluated on D4RL benchmarks. It matches or exceeds Diffusion Veteran and other diffusion planners on long-horizon domains: in Kitchen, PG achieves an average of $g_s$5 versus $g_s$6 for DV* and $g_s$7 for Hierarchical Diffuser; in AntMaze, $g_s$8 versus $g_s$9 for DV*; and in Maze2D, $$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$0 versus $$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$1 for DV* and $$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$2 for Hierarchical Diffuser. On MuJoCo tasks, it is reported as the best diffusion planner, although diffusion policies such as DQL remain slightly stronger on average. The paper also reports that performance with a latent critic is nearly identical to the variant that backpropagates through $$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$3, with much lower compute cost (2505.10881).

4. Diffusion-feature priors for dataset distillation

In dataset distillation, prior-guided distillation is formulated around the “trifecta” of diversity, generalization, and representativeness. The problem is to synthesize a compact dataset

$$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$4

with $$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$5 such that a model trained on $$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$6 generalizes similarly to one trained on the original training set $$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$7. The paper argues that diffusion models already provide strong diversity and fidelity but that existing diffusion-based dataset distillation methods under-use the representativeness prior implicit in the diffusion backbone and often require extra constraints or retraining (Su et al., 20 Oct 2025).

The proposed framework, Diffusion As Priors (DAP), uses a pretrained diffusion model in three roles: as a diversity prior via the standard score $$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$8, as a generalization prior via the regularization induced by diffusion training, and as a representativeness prior via internal features $$\nabla_{\mathbf{x}}\log p(\mathbf{x}\mid \mathcal{R}) = \nabla_{\mathbf{x}}\log p(\mathbf{x}) + \nabla_{\mathbf{x}}\log p(\mathcal{R}\mid \mathbf{x}),$$9 extracted from the backbone. Representativeness is defined by similarity in feature space using a Mercer kernel $u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$0 and the kernel-induced distance

$u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$1

The paper proves that if $u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$2 is positive semidefinite, then $u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$3 is a proper metric, and that this distance factorizes as a norm distance in feature space. In practice, the default choice is the linear kernel, so that

$u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$4

The representativeness prior is then written as an energy-based term

$u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$5

with guidance scale $u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$6 (Su et al., 20 Oct 2025).

This prior is injected directly into the reverse diffusion dynamics. The reverse SDE is modified so that the total score becomes

$u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$7

which the paper writes as an energy-based guidance term of the form

$u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$8

No diffusion weights are updated. The framework is therefore training-free in the sense that the diffusion model remains frozen and no extra classifier or projector is learned. Guidance is applied during sampling, optionally only during early denoising steps through an early-stop parameter $u_t = \mathcal{F}(u), \qquad u(x,0)=a(x),$9 (Su et al., 20 Oct 2025).

The implementation uses two diffusion backbones: DiT-XL/2-256 trained on ImageNet-1K and Stable Diffusion v1.5 pretrained on LAION. For SD, the best feature extractor is a mid-level U-Net layer; for DiT, early transformer blocks such as layers 4–12 perform best. The paper reports that moderate guidance improves downstream distillation accuracy, whereas too large a guidance scale harms diversity and generalization. Example values given are $$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$0 versus $$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$1 for DiT, and $$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$2 versus $$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$3 for SD (Su et al., 20 Oct 2025).

On ImageNet-1K with DiT under soft-label evaluation using ResNet-18, DAP reaches $$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$4 Top-1 at IPC 10 and $$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$5 at IPC 50, compared with best baselines around $$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$6–$$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$7 and $$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$8 or $$u^{n+1} = \Phi_{\Delta t}(u^n), \qquad n=0,\dots,N-1,$$9, respectively. In cross-architecture tests on ImageNet-1K, it obtains $u^N \approx u(\cdot,T)$0 versus $u^N \approx u(\cdot,T)$1 and $u^N \approx u(\cdot,T)$2 on ResNet-101 at IPC10, and $u^N \approx u(\cdot,T)$3 versus $u^N \approx u(\cdot,T)$4 and $u^N \approx u(\cdot,T)$5 on MobileNet-V2 at IPC10. The paper further reports that downsampling a distilled IPC100 set to IPC50 or IPC10 causes minimal degradation for DAP, whereas IGD and MGD$u^N \approx u(\cdot,T)$6 often lose at least $u^N \approx u(\cdot,T)$7, and that DAP’s synthetic samples align well with both train and test feature distributions in t-SNE visualizations (Su et al., 20 Oct 2025).

5. Prediction-guided detection distillation and acronym overlap

A distinct use of the acronym PGD appears in dense object detection, where it stands for Prediction-Guided Distillation rather than Prior-Guided Data Distillation. The method nonetheless instantiates a related principle: the teacher’s prediction quality acts as a prior that determines where and how strongly distillation should be applied. The quality score for a predicted box $u^N \approx u(\cdot,T)$8 relative to a ground-truth box $u^N \approx u(\cdot,T)$9 is defined from an indicator that the location lies inside $N=T/\Delta t$0, the teacher classification probability for the ground-truth class, and the IoU between predicted and ground-truth boxes. Per location, the method keeps the maximum quality over all predictions at that spatial coordinate, then selects the top-$N=T/\Delta t$1 locations per object, with $N=T/\Delta t$2 in experiments (Yang et al., 2022).

Rather than weighting these top-$N=T/\Delta t$3 locations uniformly, the method fits a 2D Gaussian by maximum likelihood:

$N=T/\Delta t$4

The resulting Gaussian exponent defines the importance of each selected location, and overlapping objects are resolved by taking the maximum importance. The foreground mask is then normalized per FPN level and used to weight both feature distillation and attention distillation. Classification and regression are distilled separately, with distinct masks and hyperparameters such as $N=T/\Delta t$5 and $N=T/\Delta t$6 (Yang et al., 2022).

The empirical claim motivating the method is that only a very small fraction of features inside a ground-truth box account for the teacher’s strong detection performance. In a masking experiment on COCO, removing the top-$N=T/\Delta t$7 of predictions by quality produces roughly a $N=T/\Delta t$8 drop in AP. The method therefore focuses distillation on “key predictive regions” rather than all foreground pixels (Yang et al., 2022).

On COCO, the paper reports between $N=T/\Delta t$9 and $\mathcal{C}$00 AP improvement when using ResNet-101 and ResNet-50 as teacher and student backbones. Detailed results include ATSS improving from $\mathcal{C}$01 AP for the student to $\mathcal{C}$02 AP under PGD, FCOS from $\mathcal{C}$03 to $\mathcal{C}$04, GFL from $\mathcal{C}$05 to $\mathcal{C}$06, and DDOD from $\mathcal{C}$07 to $\mathcal{C}$08. On CrowdHuman, the abstract reports $\mathcal{C}$09 and $\mathcal{C}$10 improvements in MR and AP, while the detailed table for DDOD shows MR improving from $\mathcal{C}$11 to $\mathcal{C}$12 and AP from $\mathcal{C}$13 to $\mathcal{C}$14 (Yang et al., 2022).

The detection work is therefore best treated as terminologically adjacent rather than identical to later “Prior-Guided Data Distillation” formulations. Its relevance lies in showing an earlier selective-distillation design in which a teacher-derived prior over spatial importance replaces uniform or purely geometry-driven supervision.

6. Limitations, design trade-offs, and research directions

The literature identifies several recurring constraints on PGD methods. In PDE operator learning, the decisive bottleneck is the differentiable numerical solver. Backpropagation through the solver dominates cost; for Navier–Stokes at $\mathcal{C}$15 timesteps, each PGD step requires roughly $\mathcal{C}$16–$\mathcal{C}$17 seconds, the backward pass is about twice the cost of the forward pass, and GPU memory on a B200 limits batch size to at most $\mathcal{C}$18. The priors are also relatively simple—primarily $\mathcal{C}$19 bounds and value clipping rather than explicit invariants such as mass, energy, or enstrophy preservation—and not all industrial solvers are differentiable or easy to port to AD frameworks (Sun, 21 Oct 2025).

In offline RL, the main limitations concern approximation and tuning. The latent-space reformulation depends on an approximate bijectivity assumption for DDIM, which the authors explicitly describe as an approximation that future flow-matching approaches might make exact. The method also introduces extra components—a prior network and a latent critic—and performance is sensitive to the regularization coefficient $\mathcal{C}$20, which is swept over $\mathcal{C}$21. Prior architecture is not deeply explored beyond a GRU-based Gaussian or Gaussian-mixture parameterization (2505.10881).

In diffusion-based dataset distillation, the dominant drawback is sampling-time overhead. Guidance requires computing diffusion features for noisy training samples and backpropagating through the frozen backbone at each guided step. The paper reports, for ImageNet-1K IPC10 on a single A40, about $\mathcal{C}$22–$\mathcal{C}$23 seconds per iteration and $\mathcal{C}$24 GB memory for SD as the data size grows from $\mathcal{C}$25 to $\mathcal{C}$26, and about $\mathcal{C}$27–$\mathcal{C}$28 seconds per iteration with $\mathcal{C}$29 GB for DiT. The method is also sensitive to the feature layer $\mathcal{C}$30, the guidance scale $\mathcal{C}$31, and the early-stop threshold $\mathcal{C}$32; overly strong guidance reduces diversity and harms downstream performance (Su et al., 20 Oct 2025).

The papers collectively point toward several future directions. In operator learning, proposed extensions include priors that preserve mass, energy budgets, or enstrophy; spectral band constraints; uncertainty-guided sampling; complex geometries; and hybrid or reduced-order solvers for cheaper differentiation (Sun, 21 Oct 2025). In offline RL, more expressive prior architectures such as transformers or flows and exact latent-space change-of-variables formulations are left open (2505.10881). In dataset distillation, non-image modalities and tasks beyond classification remain unexplored, and the broader lesson is that prior guidance should shape, not dominate, the generative dynamics (Su et al., 20 Oct 2025).

Taken together, these works establish PGD as a general strategy for concentrating distillation effort on regions favored by domain priors: physically valid perturbations in function space, behavior-regularized latent regions for planning, or representative neighborhoods in diffusion feature space. The unifying objective is not merely compression, but selective transfer of the most informative structure from a stronger model, simulator, or pretrained generative prior into a smaller and more efficient downstream system.