Dynamic Resolution Frame Sampling

• DRFS is a method that dynamically allocates sampling density and resolution across video frames to focus processing on critical content.
• It jointly optimizes frame selection and per-frame resolution allocation under a fixed compute or token budget to enhance performance.
• Empirical results show that DRFS improves action recognition and video reasoning while reducing computational overhead compared to uniform processing.

Searching arXiv for the cited DRFS-related papers to ground the article in current records. Dynamic Resolution Frame Sampling (DRFS) denotes a class of methods that allocate a limited spatiotemporal sampling or compute budget non-uniformly across video frames, image regions, or both, so that temporally important, query-relevant, or visually complex content receives higher fidelity while redundant or stable content is processed more coarsely. Across the literature, the term is used both narrowly and broadly. In recent video-LLM work, it refers explicitly to joint frame selection and per-frame token or resolution allocation (Chen et al., 12 May 2026). In related earlier work, the same underlying principle appears as adaptive foveation in single-pixel imaging (Phillips et al., 2016), per-frame resolution selection for action recognition (Meng et al., 2020), query-aware multi-resolution allocation for Video-LLMs (Zhang et al., 27 Jun 2025), dynamic sparse sensor readout (Jonscher et al., 2022), and reinforcement-learning-based training over temporal-spatial trade-offs (Ge et al., 28 Sep 2025). The unifying idea is to replace uniform processing with content-conditioned, budget-aware heterogeneity in resolution, temporal refresh, or both.

1. Conceptual scope and definitions

DRFS is most naturally defined by three coupled decisions: which visual units to process, at what fidelity to process them, and how to distribute a fixed budget across space or time. In the most explicit formulation, LDDR treats long-video understanding as a joint optimization over frame selection and per-frame visual-token allocation under a total token budget $C_{\mathrm{total}}$, rather than as frame selection alone (Chen et al., 12 May 2026). Q-Frame adopts a closely related formulation for Video-LLMs, separating the problem into Query-aware Frame Selection and Multi-Resolution Adaptation, with different selected frames assigned different spatial resolutions under a fixed token-budget-equivalent rule such as

$K + \frac{M}{4} + \frac{N}{16} = 8$

for high-, medium-, and low-resolution frame groups (Zhang et al., 27 Jun 2025).

A broader interpretation includes systems that do not use the term DRFS but embody the same resource-allocation logic. AR-Net selects a per-frame input resolution, with frame skipping treated as a “zero-resolution” action, so that each frame can receive a different compute level conditioned on context (Meng et al., 2020). In adaptive single-pixel imaging, the field of view is partitioned into a high-resolution foveal region and a lower-resolution periphery, with both resolution and effective exposure-time varying across the image and evolving with scene dynamics (Phillips et al., 2016). In dynamic non-regular imaging sensors, only a subset of pixels is read out in each frame, with complementary sparse patterns over time allowing high temporal acquisition and later full-resolution reconstruction (Jonscher et al., 2022). These systems suggest a common abstraction: DRFS is dynamic allocation of sampling density, spatial fidelity, or temporal support under a constrained acquisition or inference budget.

A source of ambiguity is that some papers adapt only temporal access rather than spatial resolution. The “Adaptive Video Understanding Agent” uses query-adaptive temporal frame sampling and feedback-driven reasoning, but it does not vary frame or patch resolution; its closest analogue to DRFS is dynamic temporal granularity rather than spatial-resolution scheduling (Jeoung et al., 2024). Conversely, “Dynamic resolution switching for live streaming” adapts spatial resolution per segment but does not alter frame rate or frame sampling (Xiong et al., 15 May 2026). The literature therefore supports a strict definition—joint temporal and spatial adaptation—and a looser one in which either dimension may be adaptive.

2. Historical precursors and early formulations

One of the clearest early precursors appears in adaptive foveated single-pixel imaging, where a high-resolution fovea follows motion while every sub-frame still covers the full field of view (Phillips et al., 2016). The paper’s central departure from uniform sampling is that every spatial location need not share the same instantaneous resolution or effective exposure time. A sub-frame may contain $M = 64 \times 64 = 4096$ high-resolution pixels in the underlying lattice but only $N = 1024$ independent cells, with the fovea represented on a regular Cartesian patch and the periphery on a lower-resolution cylindrical-polar layout (Phillips et al., 2016). This produces a tiered super-sampling regime in which static regions accumulate detail across multiple sub-frames while moving regions are refreshed using recent data only.

The mathematical reconstruction model in that work is notable because it formalizes spatially variant sensing without invoking compressive sensing as the primary inverse problem. If $\bm{s}_n = T\bm{h}_n$ denotes the stretched sensing pattern, then the spatially variant reconstruction is

$$\bm{o}_{sv} = \bm{A}^{-1} \frac{1}{N}\sum_{n=1}^N b_n \bm{s}_n$$

with $\bm{A}$ an area matrix encoding cell size (Phillips et al., 2016). Multi-frame fusion is then performed either by weighted averaging or by solving a linear-constraints least-squares system across overlapping sub-frame cells. The resulting video has both “resolution” and “effective exposure-time” that “spatially vary and adapt dynamically in response to the evolution of the scene” (Phillips et al., 2016). This is structurally close to later DRFS formulations, even though the sensing modality is not a conventional focal-plane video pipeline.

A different precursor appears in sensor design. The dynamic non-regular sampling sensor uses a modified 4-way shared-pixel readout to acquire only 25%, 50%, or 75% of pixels per frame using time-varying complementary patterns, then reconstructs full high spatial resolution with 3D frequency-selective reconstruction (Jonscher et al., 2022). Here the budget constraint is sensor throughput rather than token budget or backbone FLOPS. The acquisition pattern is dynamic across time but not content-adaptive; complementary readouts ensure that static or low-motion regions receive different observed pixels over successive frames, which substantially improves recoverability relative to static sparse readout (Jonscher et al., 2022). This suggests that dynamic variation itself, even without online policy learning, can be a powerful DRFS principle when temporal redundancy is present.

3. Adaptive resolution allocation in recognition and language-conditioned video understanding

A direct neural-network formulation of DRFS appears in AR-Net for action recognition in long untrimmed videos (Meng et al., 2020). The policy network, consisting of a lightweight feature extractor $\Phi(\cdot;\theta_\Phi)$ and an LSTM, processes each candidate frame at the lowest resolution and outputs a discrete action

$a_t \in \Omega=\{0,1,\dots,L+M-1\}$

where the first $L$ actions choose one of $K + \frac{M}{4} + \frac{N}{16} = 8$0 resolutions and the remaining $K + \frac{M}{4} + \frac{N}{16} = 8$1 actions are skip operations (Meng et al., 2020). In the main ResNet configuration, the action space includes four resolutions—$K + \frac{M}{4} + \frac{N}{16} = 8$2, $K + \frac{M}{4} + \frac{N}{16} = 8$3, $K + \frac{M}{4} + \frac{N}{16} = 8$4, $K + \frac{M}{4} + \frac{N}{16} = 8$5—mapped to ResNet-50, ResNet-34, ResNet-18, and MobileNetV2 respectively, plus skip lengths of 1, 2, or 4 frames (Meng et al., 2020). This makes frame skipping an extreme case of resolution reduction.

The policy is trained jointly with the recognizer using Gumbel-Max in the forward pass and Gumbel-Softmax in the backward pass: $K + \frac{M}{4} + \frac{N}{16} = 8$6 and a composite loss

$K + \frac{M}{4} + \frac{N}{16} = 8$7

combining classification accuracy, compute cost, and action-usage regularization (Meng et al., 2020). The key ablation for DRFS interpretation is that “Resolution Only” already outperforms both Uniform and “Skipping Only” at similar compute on ActivityNet, while “Resolution + Skipping” performs best (Meng et al., 2020). This shows that dynamic resolution is not reducible to frame dropping.

Query-conditioned video-LLM front ends refine the same idea. Q-Frame first uniformly downsamples the full video into $K + \frac{M}{4} + \frac{N}{16} = 8$8 candidates, then uses a CLIP-like text-image encoder to score frame-query relevance by inner product: $K + \frac{M}{4} + \frac{N}{16} = 8$9 converts these scores to a softmax distribution, perturbs them with Gumbel noise, and selects top-$M = 64 \times 64 = 4096$0 frames under the perturbed logits (Zhang et al., 27 Jun 2025). Multi-Resolution Adaptation then assigns selected frames to three resolution tiers according to relevance rank. The best reported allocation for Qwen2-VL is $M = 64 \times 64 = 4096$1: 4 high-resolution, 8 medium-resolution, and 32 low-resolution frames, under a token-budget-equivalent design in which one medium frame costs $M = 64 \times 64 = 4096$2 and one low frame $M = 64 \times 64 = 4096$3 of a high-resolution frame (Zhang et al., 27 Jun 2025). On LongVideoBench, this raises Qwen2-VL from 53.5 to 58.4 with only a $M = 64 \times 64 = 4096$4 increase in average tokens per video relative to an 8-frame all-high-resolution baseline (Zhang et al., 27 Jun 2025).

LDDR makes the DRFS formulation more explicit. It decodes frames at 1 FPS, extracts frame and query embeddings with LongCLIP, and builds query-conditioned features

$M = 64 \times 64 = 4096$5

so that the DPP kernel is

$M = 64 \times 64 = 4096$6

(Chen et al., 12 May 2026). Candidate frame selection is global and diversity-aware, using greedy MAP inference on $M = 64 \times 64 = 4096$7. Dynamic resolution allocation is then guided by Group DPP importance, a leave-one-out determinant contribution

$M = 64 \times 64 = 4096$8

Retained frames receive token counts $M = 64 \times 64 = 4096$9, with $N = 1024$0, $N = 1024$1, and

$N = 1024$2

for a frame-equivalent budget $N = 1024$3 (Chen et al., 12 May 2026). This is a prototypical DRFS pipeline: query-aware global selection plus budget-feasible per-frame fidelity allocation.

4. Training-time DRFS and reinforcement learning

FrameMind introduces DRFS as a training methodology rather than primarily an inference-time selector (Ge et al., 28 Sep 2025). For each video-question pair, DRFS generates $N = 1024$4 parallel visual inputs spanning a resolution ladder from low-resolution temporal scanning to high-resolution spatial focus. If $N = 1024$5 and $N = 1024$6 are the low- and high-resolution endpoints, then for ladder index $N = 1024$7,

$N = 1024$8

(Ge et al., 28 Sep 2025). Each configuration samples $N = 1024$9 frames uniformly and resizes them to $\bm{s}_n = T\bm{h}_n$0, creating a family of initial evidential states for the same question.

These ladder views are embedded in a multi-turn reasoning framework, Frame-Interleaved Chain-of-Thought, where the policy alternates between textual reasoning and tool-based visual acquisition: $\bm{s}_n = T\bm{h}_n$1 (Ge et al., 28 Sep 2025). Tools include FrameAt and VideoClip, which retrieve targeted high-resolution frames or interval clips during reasoning. DRFS shapes the policy by making it robust to different initial trade-offs between temporal breadth and spatial detail.

Learning uses DRFS-GRPO, which groups the $\bm{s}_n = T\bm{h}_n$2 rollouts from the same training example and computes a group-relative advantage

$\bm{s}_n = T\bm{h}_n$3

(Ge et al., 28 Sep 2025). A PPO-style objective then reinforces actions from higher-performing ladder views without requiring frame-level supervision. The ablation on VideoMME is central: fixed-sampling GRPO at 32 evaluation frames yields 54.0 overall and 49.5 on long videos, whereas DRFS-32 achieves 60.9 overall and 57.5 on long videos (Ge et al., 28 Sep 2025). This suggests that exposure to multiple temporal-spatial regimes during training materially improves learned perception policies.

A plausible implication is that DRFS can function not only as an online scheduler but also as a curriculum over evidential uncertainty. In FrameMind, the model learns under systematically varied observation regimes before it ever decides where to inspect next. That differs from training-free front ends such as Q-Frame (Zhang et al., 27 Jun 2025) or LDDR (Chen et al., 12 May 2026), but addresses the same underlying question: how should a system reason when the visual budget is insufficient to show everything at once?

5. Mathematical structure and optimization patterns

Despite differences in modality, several recurring mathematical motifs organize DRFS methods.

First is explicit budgeted allocation. In AR-Net, compute is modeled through a FLOPS lookup table and penalized in expectation: $\bm{s}_n = T\bm{h}_n$4 (Meng et al., 2020). In Q-Frame, the budget is imposed through fixed token-equivalent rules such as

$\bm{s}_n = T\bm{h}_n$5

(Zhang et al., 27 Jun 2025). In LDDR, the budget is the total visual-token count $\bm{s}_n = T\bm{h}_n$6, with per-frame allocations clipped to $\bm{s}_n = T\bm{h}_n$7 tokens and selected by a largest-feasible-prefix search (Chen et al., 12 May 2026). In the adaptive single-pixel system, the budget is the number of sequential correlation measurements, fixed at $\bm{s}_n = T\bm{h}_n$8 cells per sub-frame, while detail is redistributed spatially and temporally (Phillips et al., 2016).

Second is a separation between candidate generation and adaptive composition. Q-Frame first forms a candidate pool by uniform downsampling, then performs query-aware selection and rank-based resolution assignment (Zhang et al., 27 Jun 2025). LDDR first selects a diverse, query-conditioned candidate set via DPP, then computes Group DPP importance to decide retention and resolution (Chen et al., 12 May 2026). In single-pixel imaging, spatially variant sub-frames are acquired first, then fused through weighted averaging or linear constraints, with stale regions deleted from the system when change is detected (Phillips et al., 2016). This suggests a general DRFS architecture in which initial acquisition is broad but coarse, while later steps refine or reallocate support.

Third is a reliance on diversity or complementarity rather than relevance alone. In LDDR, DPP selection maximizes $\bm{s}_n = T\bm{h}_n$9, penalizing redundancy by geometry in query-conditioned feature space (Chen et al., 12 May 2026). In Q-Frame, Gumbel-Max top-$$\bm{o}_{sv} = \bm{A}^{-1} \frac{1}{N}\sum_{n=1}^N b_n \bm{s}_n$$0 introduces stochastic exploration beyond pure CLIP top-$$\bm{o}_{sv} = \bm{A}^{-1} \frac{1}{N}\sum_{n=1}^N b_n \bm{s}_n$$1 matching, which empirically improves over deterministic CLIP ranking (Zhang et al., 27 Jun 2025). In dynamic non-regular sensors, complementary sparse readout patterns ensure that different pixel locations are observed over successive frames (Jonscher et al., 2022). These approaches differ technically, but all indicate that DRFS is strongest when it avoids spending budget on repeated evidence.

6. Empirical behavior, applications, and limitations

The empirical case for DRFS is strongest in budget-constrained settings. AR-Net on ActivityNet-v1.3 improves Uniform processing from 72.5 mAP at 65.76 GFLOPS/video to 73.8 mAP at 33.47 GFLOPS/video in the ResNet variant, and to 79.7 mAP at 15.29 GFLOPS/video in the EfficientNet variant (Meng et al., 2020). Q-Frame raises Qwen2-VL from 56.9 to 65.4 on MLVU and from 53.5 to 58.4 on LongVideoBench in the fixed-token setting with the $$\bm{o}_{sv} = \bm{A}^{-1} \frac{1}{N}\sum_{n=1}^N b_n \bm{s}_n$$2 multi-resolution configuration (Zhang et al., 27 Jun 2025). LDDR reports gains of about 2.5 points over the next-best baselines under budget-constrained settings and 1.6 points in higher-budget scenarios, with the advantage most visible on long videos and small budgets (Chen et al., 12 May 2026). FrameMind’s DRFS ablation shows especially large improvements on long-form VideoMME splits (Ge et al., 28 Sep 2025).

The application scope is correspondingly broad. In video understanding, DRFS serves action recognition (Meng et al., 2020), video question answering and reasoning (Zhang et al., 27 Jun 2025, Chen et al., 12 May 2026, Ge et al., 28 Sep 2025), and long-video agentic exploration (Jeoung et al., 2024). In computational imaging, it addresses sequential-measurement bottlenecks in single-pixel cameras (Phillips et al., 2016) and throughput-limited sensor readout (Jonscher et al., 2022). A plausible implication is that DRFS is most natural wherever the acquisition or inference pipeline is bottlenecked by a sequential, tokenized, or bandwidth-limited resource rather than by uniform full-frame access.

Several limitations recur. Query-conditioned methods depend heavily on the quality of the retrieval encoder; Q-Frame’s ablations show Long-CLIP outperforming CLIP and SigLIP, and LDDR likewise depends on LongCLIP features for effective global DPP selection (Zhang et al., 27 Jun 2025, Chen et al., 12 May 2026). Dynamic resolution alone is not sufficient: in Q-Frame, MRA with uniform sampling underperforms the baseline, whereas QFS + MRA is best (Zhang et al., 27 Jun 2025). Temporal continuity is often weakly modeled: Q-Frame explicitly notes failure on reasoning tasks that require event sequence or causal reconstruction because it ranks frames by semantic relevance rather than temporal structure (Zhang et al., 27 Jun 2025). Single-pixel and sensor-side methods assume local temporal constancy; when motion is rapid, stale or inconsistent information contaminates reconstruction, and gains over static readout can disappear (Phillips et al., 2016, Jonscher et al., 2022). Training-free front ends are deployable but cannot exploit task loss directly, whereas RL-based approaches are more adaptive but more complex to train (Ge et al., 28 Sep 2025).

A common misconception is that DRFS is equivalent to frame skipping. The literature does not support that reduction. AR-Net shows “Resolution Only” outperforming “Skipping Only” at similar cost (Meng et al., 2020). Q-Frame’s best configuration uses a mixture of high-, medium-, and low-resolution frames rather than only more low-resolution frames or fewer high-resolution ones (Zhang et al., 27 Jun 2025). LDDR’s central claim is that even among already selected informative frames, not all deserve the same visual-token budget (Chen et al., 12 May 2026). DRFS is therefore better characterized as graded budget allocation than as binary keep-versus-drop selection.

7. Relation to adjacent concepts and terminological cautions

DRFS overlaps with, but should be distinguished from, several adjacent areas. It is not identical to compressive sensing: the adaptive single-pixel paper explicitly frames its method as deterministic sampling with direct linear reconstruction or least-squares fusion, with compressive sensing only as prior art or a complementary tool (Phillips et al., 2016). It is not identical to adaptive bitrate streaming: Dynamic Resolution Switching for live streaming chooses the best spatial resolution per segment at a fixed bitrate using a bitstream-based VQM, but does not adapt frame rate or frame sampling (Xiong et al., 15 May 2026). It is not the same as purely temporal frame access systems such as the Adaptive Video Understanding Agent, which changes what frames are inspected and how densely in time, but not their spatial fidelity (Jeoung et al., 2024).

The acronym itself is also overloaded. “DRFS” in “Delta Rectified Flow Sampling for Text-to-Image Editing” denotes Delta Rectified Flow Sampling, an inversion-free text-to-image editing method for rectified flow models, unrelated to dynamic resolution or video/frame sampling (Beaudouin et al., 1 Sep 2025). Any encyclopedia treatment therefore requires domain qualification: in video and computational imaging, DRFS refers to dynamic resolution frame sampling or closely related spatiotemporal budget allocation; in generative modeling, the same acronym can refer to a different method entirely (Beaudouin et al., 1 Sep 2025).

The most defensible synthesis across the cited work is that DRFS is a design pattern rather than a single algorithmic family. Its essential ingredients are: a constrained acquisition or inference budget; heterogeneous allocation of that budget across time, space, or frames; and a control signal based on relevance, diversity, motion, detail density, or utility. The specific implementation may be deterministic and hardware-structured (Jonscher et al., 2022), query-aware and training-free (Zhang et al., 27 Jun 2025, Chen et al., 12 May 2026), recurrent and cost-regularized (Meng et al., 2020), or reinforcement-learned over a resolution ladder (Ge et al., 28 Sep 2025). What unifies them is the rejection of uniform fidelity as the default operating point under scarcity.