---
title: Dynamic Resolution Frame Sampling
url: https://www.emergentmind.com/topics/dynamic-resolution-frame-sampling-drfs
type: topic
---

# Dynamic Resolution Frame Sampling

Searching arXiv for the cited DRFS-related papers to ground the article in current records.
Dynamic Resolution Frame Sampling (DRFS) denotes a class of methods that allocate a limited spatiotemporal sampling or compute budget non-uniformly across video frames, image regions, or both, so that temporally important, query-relevant, or visually complex content receives higher fidelity while redundant or stable content is processed more coarsely. Across the literature, the term is used both narrowly and broadly. In recent video-LLM work, it refers explicitly to joint frame selection and per-frame token or resolution allocation [2605.11477]. In related earlier work, the same underlying principle appears as adaptive foveation in single-pixel imaging [1607.08236], per-frame resolution selection for action recognition [2007.15796], query-aware multi-resolution allocation for Video-LLMs [2506.22139], dynamic sparse sensor readout [2204.03268], and reinforcement-learning-based training over temporal-spatial trade-offs [2509.24008]. The unifying idea is to replace uniform processing with content-conditioned, budget-aware heterogeneity in resolution, temporal refresh, or both.

## 1. Conceptual scope and definitions

DRFS is most naturally defined by three coupled decisions: which visual units to process, at what fidelity to process them, and how to distribute a fixed budget across space or time. In the most explicit formulation, LDDR treats long-video understanding as a joint optimization over frame selection and per-frame visual-token allocation under a total token budget \(C_{\mathrm{total}}\), rather than as frame selection alone [2605.11477]. Q-Frame adopts a closely related formulation for Video-LLMs, separating the problem into Query-aware Frame Selection and Multi-Resolution Adaptation, with different selected frames assigned different spatial resolutions under a fixed token-budget-equivalent rule such as
\[
K + \frac{M}{4} + \frac{N}{16} = 8
\]
for high-, medium-, and low-resolution frame groups [2506.22139].

A broader interpretation includes systems that do not use the term DRFS but embody the same resource-allocation logic. AR-Net selects a per-frame input resolution, with frame skipping treated as a “zero-resolution” action, so that each frame can receive a different compute level conditioned on context [2007.15796]. In adaptive single-pixel imaging, the field of view is partitioned into a high-resolution foveal region and a lower-resolution periphery, with both resolution and effective exposure-time varying across the image and evolving with scene dynamics [1607.08236]. In dynamic non-regular imaging sensors, only a subset of pixels is read out in each frame, with complementary sparse patterns over time allowing high temporal acquisition and later full-resolution reconstruction [2204.03268]. These systems suggest a common abstraction: DRFS is dynamic allocation of sampling density, spatial fidelity, or temporal support under a constrained acquisition or inference budget.

A source of ambiguity is that some papers adapt only temporal access rather than spatial resolution. The “Adaptive Video Understanding Agent” uses query-adaptive temporal frame sampling and feedback-driven reasoning, but it does not vary frame or patch resolution; its closest analogue to DRFS is dynamic temporal granularity rather than spatial-resolution scheduling [2410.20252]. Conversely, “Dynamic resolution switching for live streaming” adapts spatial resolution per segment but does not alter frame rate or frame sampling [2605.15490]. The literature therefore supports a strict definition—joint temporal and spatial adaptation—and a looser one in which either dimension may be adaptive.

## 2. Historical precursors and early formulations

One of the clearest early precursors appears in adaptive foveated single-pixel imaging, where a high-resolution fovea follows motion while every sub-frame still covers the full field of view [1607.08236]. The paper’s central departure from uniform sampling is that every spatial location need not share the same instantaneous resolution or effective exposure time. A sub-frame may contain \(M = 64 \times 64 = 4096\) high-resolution pixels in the underlying lattice but only \(N = 1024\) independent cells, with the fovea represented on a regular Cartesian patch and the periphery on a lower-resolution cylindrical-polar layout [1607.08236]. This produces a tiered super-sampling regime in which static regions accumulate detail across multiple sub-frames while moving regions are refreshed using recent data only.

The mathematical reconstruction model in that work is notable because it formalizes spatially variant sensing without invoking compressive sensing as the primary inverse problem. If \(\bm{s}_n = T\bm{h}_n\) denotes the stretched sensing pattern, then the spatially variant reconstruction is
\[
\bm{o}_{sv} = \bm{A}^{-1} \frac{1}{N}\sum_{n=1}^N b_n \bm{s}_n
\]
with \(\bm{A}\) an area matrix encoding cell size [1607.08236]. Multi-frame fusion is then performed either by weighted averaging or by solving a linear-constraints least-squares system across overlapping sub-frame cells. The resulting video has both “resolution” and “effective exposure-time” that “spatially vary and adapt dynamically in response to the evolution of the scene” [1607.08236]. This is structurally close to later DRFS formulations, even though the sensing modality is not a conventional focal-plane video pipeline.

A different precursor appears in sensor design. The dynamic non-regular sampling sensor uses a modified 4-way shared-pixel readout to acquire only 25%, 50%, or 75% of pixels per frame using time-varying complementary patterns, then reconstructs full high spatial resolution with 3D frequency-selective reconstruction [2204.03268]. Here the budget constraint is sensor throughput rather than token budget or backbone FLOPS. The acquisition pattern is dynamic across time but not content-adaptive; complementary readouts ensure that static or low-motion regions receive different observed pixels over successive frames, which substantially improves recoverability relative to static sparse readout [2204.03268]. This suggests that dynamic variation itself, even without online policy learning, can be a powerful DRFS principle when temporal redundancy is present.

## 3. Adaptive resolution allocation in recognition and language-conditioned video understanding

A direct neural-network formulation of DRFS appears in AR-Net for action recognition in long untrimmed videos [2007.15796]. The policy network, consisting of a lightweight feature extractor \(\Phi(\cdot;\theta_\Phi)\) and an LSTM, processes each candidate frame at the lowest resolution and outputs a discrete action
\[
a_t \in \Omega=\{0,1,\dots,L+M-1\}
\]
where the first \(L\) actions choose one of \(L\) resolutions and the remaining \(M\) actions are skip operations [2007.15796]. In the main ResNet configuration, the action space includes four resolutions—\(224\times224\), \(168\times168\), \(112\times112\), \(84\times84\)—mapped to ResNet-50, ResNet-34, ResNet-18, and MobileNetV2 respectively, plus skip lengths of 1, 2, or 4 frames [2007.15796]. This makes frame skipping an extreme case of resolution reduction.

The policy is trained jointly with the recognizer using Gumbel-Max in the forward pass and Gumbel-Softmax in the backward pass:
\[
P_i=\frac{\exp((\log p_i+G_i)/\tau)}{\sum_{j=0}^{L+M-1} \exp((\log p_j+G_j)/\tau)}
\]
and a composite loss
\[
\mathcal{L}=(1-\alpha)\cdot \mathcal{L}_{acc}+ \alpha \cdot\mathcal{L}_{flops}+\beta \cdot \mathcal{L}_{uni}
\]
combining classification accuracy, compute cost, and action-usage regularization [2007.15796]. The key ablation for DRFS interpretation is that “Resolution Only” already outperforms both Uniform and “Skipping Only” at similar compute on ActivityNet, while “Resolution + Skipping” performs best [2007.15796]. This shows that dynamic resolution is not reducible to frame dropping.

Query-conditioned video-LLM front ends refine the same idea. Q-Frame first uniformly downsamples the full video into \(T\) candidates, then uses a CLIP-like text-image encoder to score frame-query relevance by inner product:
\[
I = QF^{\text{T}}
\]
converts these scores to a softmax distribution, perturbs them with Gumbel noise, and selects top-\(K\) frames under the perturbed logits [2506.22139]. Multi-Resolution Adaptation then assigns selected frames to three resolution tiers according to relevance rank. The best reported allocation for Qwen2-VL is \(4+8+32\): 4 high-resolution, 8 medium-resolution, and 32 low-resolution frames, under a token-budget-equivalent design in which one medium frame costs \(1/4\) and one low frame \(1/16\) of a high-resolution frame [2506.22139]. On LongVideoBench, this raises Qwen2-VL from 53.5 to 58.4 with only a \(+3.4\%\) increase in average tokens per video relative to an 8-frame all-high-resolution baseline [2506.22139].

LDDR makes the DRFS formulation more explicit. It decodes frames at 1 FPS, extracts frame and query embeddings with LongCLIP, and builds query-conditioned features
\[
\mathbf \Phi=\mathrm{diag}(\tilde{\mathbf r})\hat{\mathbf F}
\]
so that the DPP kernel is
\[
\mathbf L=\mathrm{diag}(\tilde{\mathbf r})\hat{\mathbf F}\hat{\mathbf F}^\top \mathrm{diag}(\tilde{\mathbf r})=\mathbf \Phi\mathbf \Phi^\top
\]
[2605.11477]. Candidate frame selection is global and diversity-aware, using greedy MAP inference on \(\log\det(L_S)\). Dynamic resolution allocation is then guided by Group DPP importance, a leave-one-out determinant contribution
\[
\mathcal I_t = \frac{ \det(\mathbf G_{\mathcal S}) }{ \det(\mathbf G_{\mathcal S\setminus\{t\}) }.
\]
Retained frames receive token counts \(w_t\in[w_{\min},w_{\max}]\), with \(w_{\min}=256\), \(w_{\max}=1024\), and
\[
C_{\mathrm{total} = F \times 1024
\]
for a frame-equivalent budget \(F\) [2605.11477]. This is a prototypical DRFS pipeline: query-aware global selection plus budget-feasible per-frame fidelity allocation.

## 4. Training-time DRFS and reinforcement learning

FrameMind introduces DRFS as a training methodology rather than primarily an inference-time selector [2509.24008]. For each video-question pair, DRFS generates \(G\) parallel visual inputs spanning a resolution ladder from low-resolution temporal scanning to high-resolution spatial focus. If \((N_L,H_L,W_L)\) and \((N_H,H_H,W_H)\) are the low- and high-resolution endpoints, then for ladder index \(g\),
\[
r=\frac{g-1}{G-1},
\qquad
N_g = (1-r)N_L + rN_H,
\qquad
(H_g, W_g) = (1-r)(H_L, W_L) + r(H_H, W_H)
\]
[2509.24008]. Each configuration samples \(N_g\) frames uniformly and resizes them to \(H_g \times W_g\), creating a family of initial evidential states for the same question.

These ladder views are embedded in a multi-turn reasoning framework, Frame-Interleaved Chain-of-Thought, where the policy alternates between textual reasoning and tool-based visual acquisition:
\[
(T_k,\, C_k) \sim \pi_\theta(\cdot \mid H_{k-1},\, E_{k-1})
\]
[2509.24008]. Tools include **FrameAt** and **VideoClip**, which retrieve targeted high-resolution frames or interval clips during reasoning. DRFS shapes the policy by making it robust to different initial trade-offs between temporal breadth and spatial detail.

Learning uses DRFS-GRPO, which groups the \(G\) rollouts from the same training example and computes a group-relative advantage
\[
\bar{R}_i = \frac{1}{G}\sum_g R_i^{(g)},
\qquad
A_i^{(g)} = R_i^{(g)} - \bar{R}_i
\]
[2509.24008]. A PPO-style objective then reinforces actions from higher-performing ladder views without requiring frame-level supervision. The ablation on VideoMME is central: fixed-sampling GRPO at 32 evaluation frames yields 54.0 overall and 49.5 on long videos, whereas DRFS-32 achieves 60.9 overall and 57.5 on long videos [2509.24008]. This suggests that exposure to multiple temporal-spatial regimes during training materially improves learned perception policies.

A plausible implication is that DRFS can function not only as an online scheduler but also as a curriculum over evidential uncertainty. In FrameMind, the model learns under systematically varied observation regimes before it ever decides where to inspect next. That differs from training-free front ends such as Q-Frame [2506.22139] or LDDR [2605.11477], but addresses the same underlying question: how should a system reason when the visual budget is insufficient to show everything at once?

## 5. Mathematical structure and optimization patterns

Despite differences in modality, several recurring mathematical motifs organize DRFS methods.

First is explicit budgeted allocation. In AR-Net, compute is modeled through a FLOPS lookup table and penalized in expectation:
\[
\mathcal{L}_{flops}=\E_{(V,y)\sim \mathcal{D}_{train}\left[\frac{1}{T}\sum\limits_{t=0}^{T-1} \text{FLOPS}_{\mathcal{F}(a_{V,t})}\right]
\]
[2007.15796]. In Q-Frame, the budget is imposed through fixed token-equivalent rules such as
\[
K + \frac{M}{4} + \frac{N}{16} = 8
\]
[2506.22139]. In LDDR, the budget is the total visual-token count \(C_{\mathrm{total}}\), with per-frame allocations clipped to \([256,1024]\) tokens and selected by a largest-feasible-prefix search [2605.11477]. In the adaptive single-pixel system, the budget is the number of sequential correlation measurements, fixed at \(N=1024\) cells per sub-frame, while detail is redistributed spatially and temporally [1607.08236].

Second is a separation between candidate generation and adaptive composition. Q-Frame first forms a candidate pool by uniform downsampling, then performs query-aware selection and rank-based resolution assignment [2506.22139]. LDDR first selects a diverse, query-conditioned candidate set via DPP, then computes Group DPP importance to decide retention and resolution [2605.11477]. In single-pixel imaging, spatially variant sub-frames are acquired first, then fused through weighted averaging or linear constraints, with stale regions deleted from the system when change is detected [1607.08236]. This suggests a general DRFS architecture in which initial acquisition is broad but coarse, while later steps refine or reallocate support.

Third is a reliance on diversity or complementarity rather than relevance alone. In LDDR, DPP selection maximizes \(\log\det(L_S)\), penalizing redundancy by geometry in query-conditioned feature space [2605.11477]. In Q-Frame, Gumbel-Max top-\(K\) introduces stochastic exploration beyond pure CLIP top-\(K\) matching, which empirically improves over deterministic CLIP ranking [2506.22139]. In dynamic non-regular sensors, complementary sparse readout patterns ensure that different pixel locations are observed over successive frames [2204.03268]. These approaches differ technically, but all indicate that DRFS is strongest when it avoids spending budget on repeated evidence.

## 6. Empirical behavior, applications, and limitations

The empirical case for DRFS is strongest in budget-constrained settings. AR-Net on ActivityNet-v1.3 improves Uniform processing from 72.5 mAP at 65.76 GFLOPS/video to 73.8 mAP at 33.47 GFLOPS/video in the ResNet variant, and to 79.7 mAP at 15.29 GFLOPS/video in the EfficientNet variant [2007.15796]. Q-Frame raises Qwen2-VL from 56.9 to 65.4 on MLVU and from 53.5 to 58.4 on LongVideoBench in the fixed-token setting with the \(4+8+32\) multi-resolution configuration [2506.22139]. LDDR reports gains of about 2.5 points over the next-best baselines under budget-constrained settings and 1.6 points in higher-budget scenarios, with the advantage most visible on long videos and small budgets [2605.11477]. FrameMind’s DRFS ablation shows especially large improvements on long-form VideoMME splits [2509.24008].

The application scope is correspondingly broad. In video understanding, DRFS serves action recognition [2007.15796], video question answering and reasoning [2506.22139, 2605.11477, 2509.24008], and long-video agentic exploration [2410.20252]. In computational imaging, it addresses sequential-measurement bottlenecks in single-pixel cameras [1607.08236] and throughput-limited sensor readout [2204.03268]. A plausible implication is that DRFS is most natural wherever the acquisition or inference pipeline is bottlenecked by a sequential, tokenized, or bandwidth-limited resource rather than by uniform full-frame access.

Several limitations recur. Query-conditioned methods depend heavily on the quality of the retrieval encoder; Q-Frame’s ablations show Long-CLIP outperforming CLIP and SigLIP, and LDDR likewise depends on LongCLIP features for effective global DPP selection [2506.22139, 2605.11477]. Dynamic resolution alone is not sufficient: in Q-Frame, MRA with uniform sampling underperforms the baseline, whereas QFS + MRA is best [2506.22139]. Temporal continuity is often weakly modeled: Q-Frame explicitly notes failure on reasoning tasks that require event sequence or causal reconstruction because it ranks frames by semantic relevance rather than temporal structure [2506.22139]. Single-pixel and sensor-side methods assume local temporal constancy; when motion is rapid, stale or inconsistent information contaminates reconstruction, and gains over static readout can disappear [1607.08236, 2204.03268]. Training-free front ends are deployable but cannot exploit task loss directly, whereas RL-based approaches are more adaptive but more complex to train [2509.24008].

A common misconception is that DRFS is equivalent to frame skipping. The literature does not support that reduction. AR-Net shows “Resolution Only” outperforming “Skipping Only” at similar cost [2007.15796]. Q-Frame’s best configuration uses a mixture of high-, medium-, and low-resolution frames rather than only more low-resolution frames or fewer high-resolution ones [2506.22139]. LDDR’s central claim is that even among already selected informative frames, not all deserve the same visual-token budget [2605.11477]. DRFS is therefore better characterized as graded budget allocation than as binary keep-versus-drop selection.

## 7. Relation to adjacent concepts and terminological cautions

DRFS overlaps with, but should be distinguished from, several adjacent areas. It is not identical to compressive sensing: the adaptive single-pixel paper explicitly frames its method as deterministic sampling with direct linear reconstruction or least-squares fusion, with compressive sensing only as prior art or a complementary tool [1607.08236]. It is not identical to adaptive bitrate streaming: Dynamic Resolution Switching for live streaming chooses the best spatial resolution per segment at a fixed bitrate using a bitstream-based VQM, but does not adapt frame rate or frame sampling [2605.15490]. It is not the same as purely temporal frame access systems such as the Adaptive Video Understanding Agent, which changes what frames are inspected and how densely in time, but not their spatial fidelity [2410.20252].

The acronym itself is also overloaded. “DRFS” in “Delta Rectified Flow Sampling for Text-to-Image Editing” denotes Delta Rectified Flow Sampling, an inversion-free text-to-image editing method for rectified flow models, unrelated to dynamic resolution or video/frame sampling [2509.05342]. Any encyclopedia treatment therefore requires domain qualification: in video and computational imaging, DRFS refers to dynamic resolution frame sampling or closely related spatiotemporal budget allocation; in generative modeling, the same acronym can refer to a different method entirely [2509.05342].

The most defensible synthesis across the cited work is that DRFS is a design pattern rather than a single algorithmic family. Its essential ingredients are: a constrained acquisition or inference budget; heterogeneous allocation of that budget across time, space, or frames; and a control signal based on relevance, diversity, motion, detail density, or utility. The specific implementation may be deterministic and hardware-structured [2204.03268], query-aware and training-free [2506.22139, 2605.11477], recurrent and cost-regularized [2007.15796], or reinforcement-learned over a resolution ladder [2509.24008]. What unifies them is the rejection of uniform fidelity as the default operating point under scarcity.

Source: https://www.emergentmind.com/topics/dynamic-resolution-frame-sampling-drfs