AnyFlow: Continuous Mapping in Vision

• AnyFlow is a dual-concept framework applying continuous mapping techniques to enable arbitrary-scale optical flow estimation and any-step video diffusion.
• The optical flow variant incorporates an implicit neural representation, multi-scale feature warping, and dynamic correlation lookup to capture fine motion details under varied resolutions.
• The video diffusion system distills flow-map transitions over arbitrary time intervals, reducing discretization error and ensuring consistent generative sampling.

Searching arXiv for the two "AnyFlow" papers to ground the article in the cited literature. “AnyFlow” denotes two distinct research systems that share a name but address different technical problems. In optical flow estimation, AnyFlow is a network that “for the first time, treats the flow field itself as a continuous function of image coordinates,” enabling accurate motion prediction at arbitrary spatial scales from low-resolution inputs (Jung et al., 2023). In video generation, AnyFlow is “the first any-step video diffusion distillation framework based on flow maps,” designed to support arbitrary inference budgets while preserving the test-time scaling behavior of probability-flow ODE sampling (Gu et al., 13 May 2026). The shared term therefore refers not to a single unified framework, but to two separate lines of work centered on continuity across scales or time intervals.

1. Name and scope

The 2023 AnyFlow paper addresses optical flow under input resizing, especially the degradation of small-object and boundary accuracy when images are downsized for efficiency (Jung et al., 2023). Its central claim is that optical flow can be represented as a continuous coordinate-based function and queried at arbitrary output resolutions. The method is built on a RAFT-style iterative refinement backbone augmented with an implicit neural representation, multi-scale feature warping, and dynamic correlation lookup.

The 2026 AnyFlow paper addresses few-step and any-step video diffusion distillation (Gu et al., 13 May 2026). Its starting point is the observation that consistency-distilled video models often degrade when more sampling steps are used at test time. AnyFlow replaces endpoint-only consistency with flow-map transition learning over arbitrary time intervals, together with on-policy flow map distillation.

This suggests that the common semantic thread behind the two systems is not task overlap but a methodological emphasis on continuous mappings: spatial continuity in the optical-flow variant and temporal-transition continuity in the diffusion variant.

2. AnyFlow for arbitrary-scale optical flow

In the optical-flow formulation, the flow field is modeled as a continuous function

$$f^s = \mathbb{I}(I_1, I_2), \qquad f^s \in \mathbb{R}^{sH\times sW\times 2},$$

where $s$ is any positive scale factor, with $s<1$ for downsize and $s>1$ for upsample (Jung et al., 2023). The method takes a low-resolution image pair as input and can directly produce flow at the original resolution, a down-sampled resolution, or a super-resolved resolution without relying on naive interpolation.

After the encoder and GRU update inherited from RAFT produce a hidden state $M\in\mathbb{R}^{H/8\times W/8\times C}$ and an accumulated low-resolution flow $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$, AnyFlow upsamples $f_i$ to an arbitrary output resolution via a small MLP $f_\theta$ (Jung et al., 2023). For a continuous query coordinate $x_q=(u,v)$ in $[0,H/8]\times[0,W/8]$, the network locates its nine nearest integer neighbors in the coarse flow grid, identifies the nearest feature $s$0 at integer coordinate $s$1, and feeds the tuple

$s$2

into the MLP, where $s$3 is the positional encoding from LIIF. The output

$s$4

is interpreted as convex weights on the $s$5 local neighbors of $s$6 together with an arrangement that simultaneously produces an $s$7 patch of higher-resolution flow (Jung et al., 2023). By sampling $s$8 query points and assembling patches, the method reconstructs $s$9 at resolution $s<1$0 in one shot.

A key consequence is that $s<1$1 and $s<1$2 may be chosen arbitrarily, including non-integer scalings, without changing the model (Jung et al., 2023). The paper presents this as a departure from fixed-grid optical flow prediction.

3. Optical-flow architecture, training, and inference

AnyFlow extends RAFT’s iterative design with multi-scale feature warping (Jung et al., 2023). Whereas RAFT warps only at $s<1$3 scale, AnyFlow also extracts features $s<1$4 and $s<1$5, in addition to $s<1$6. After each GRU iteration $s<1$7, the current flow estimate is upsampled via the implicit upsampler to obtain $s<1$8 and $s<1$9, and second-frame features are warped:

$s>1$0

These warped features are concatenated with first-frame features, processed by a $s>1$1 convolution, and then PixelShuffle is used to bring them back to the $s>1$2 grid:

$s>1$3

$s>1$4

$s>1$5

According to the paper, this injects high-frequency spatial cues into the update and improves boundary localization and tiny-object capture (Jung et al., 2023).

The method also replaces RAFT’s fixed-radius correlation lookup with a dynamic lookup strategy (Jung et al., 2023). At each iteration and for each pixel, a residual radius $s>1$6 is predicted and accumulated as $s>1$7. The local search window therefore grows or shrinks per pixel depending on motion magnitude, while the number of sampled points remains $s>1$8. To address blind spots when $s>1$9 becomes large, the model defines nine auxiliary sub-pixel offsets around each sample location, collects their correlation values, and feeds them into a small MLP $M\in\mathbb{R}^{H/8\times W/8\times C}$0 together with $M\in\mathbb{R}^{H/8\times W/8\times C}$1. The paper terms this “region encoding.”

Training follows the original RAFT loss:

$M\in\mathbb{R}^{H/8\times W/8\times C}$2

In all experiments, $M\in\mathbb{R}^{H/8\times W/8\times C}$3 updates during training and $M\in\mathbb{R}^{H/8\times W/8\times C}$4, and no additional photometric or smoothness terms are used (Jung et al., 2023). To encourage robustness to arbitrary scales, with probability $M\in\mathbb{R}^{H/8\times W/8\times C}$5 the input image pair is downsampled by an independently and uniformly chosen scale $M\in\mathbb{R}^{H/8\times W/8\times C}$6, typically down to 50%–90% of the original, while the model is required to recover the flow at the original resolution.

At inference, the procedure is explicit: optionally downsample the original frames; encode multi-scale features; build $M\in\mathbb{R}^{H/8\times W/8\times C}$7-scale all-pairs correlations; initialize $M\in\mathbb{R}^{H/8\times W/8\times C}$8 and $M\in\mathbb{R}^{H/8\times W/8\times C}$9; iterate GRU updates and dynamic correlation sampling; upsample each $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$0 to arbitrary scale $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$1 via $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$2; and return the final upsampled flow (Jung et al., 2023). The implicit upsampler $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$3 and region-encoding module $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$4 add approximately $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$5 M parameters beyond RAFT’s $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$6 M, and the runtime overhead is described as small.

4. Empirical profile of the optical-flow model

The optical-flow AnyFlow is reported to establish “a new state-of-the-art performance of cross-dataset generalization on the KITTI dataset,” while achieving comparable benchmark accuracy to other SOTA methods (Jung et al., 2023). In cross-dataset generalization experiments trained on FlyingChairs+Things only, RAFT achieves EPE $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$7 on Sintel-clean, while AnyFlow (dynamic) reduces this to $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$8, and region encoding further reduces it to $f_i\in\mathbb{R}^{H/8\times W/8\times 2}$9 (Jung et al., 2023). On KITTI-train, RAFT has F1-epe $f_i$0 and F1-all $f_i$1; AnyFlow (dynamic) achieves $f_i$2 EPE and $f_i$3 outliers, and region encoding yields $f_i$4.

After fine-tuning, the paper reports that on the public Sintel test set AnyFlow (dynamic) scores $f_i$5 EPE on “clean” and $f_i$6 on “final,” compared with RAFT’s $f_i$7 and $f_i$8 respectively (Jung et al., 2023). On KITTI-test, AnyFlow (dynamic) records $f_i$9 F1-all versus $f_\theta$0 for RAFT. The paper characterizes this as second best overall and top among non-Transformer, non-ImageNet-pretrained methods.

A major emphasis is robustness to input downsampling (Jung et al., 2023). When methods are fed inputs downsampled by 50%–90% and evaluated after re-upsampling flow to original size, RAFT’s Sintel-clean EPE rises from $f_\theta$1 at 100% to $f_\theta$2 at 50%, while AnyFlow rises from $f_\theta$3 to $f_\theta$4. On KITTI, RAFT rises from $f_\theta$5 to $f_\theta$6, whereas AnyFlow rises from $f_\theta$7 to $f_\theta$8. The paper interprets this stability as evidence for the advantage of continuous flow representation.

Qualitatively, the reported comparisons indicate that when inputs are halved or lower, RAFT, GMA, and GMFlow lose fine boundaries around wheels, limbs, or thin rods, and small motions vanish entirely, whereas AnyFlow retains crisp object outlines and captures tiny displacements (Jung et al., 2023). A plausible implication is that the model is especially relevant in resource-constrained settings where aggressive input resizing is operationally necessary.

5. AnyFlow for any-step video diffusion

In the 2026 work, AnyFlow denotes a video diffusion distillation framework based on flow maps rather than endpoint consistency (Gu et al., 13 May 2026). A flow map model parameterized by $f_\theta$9 implements

$x_q=(u,v)$0

with $x_q=(u,v)$1. Endpoint consistency, written as $x_q=(u,v)$2, is treated as the special case $x_q=(u,v)$3.

The underlying teacher model is described by the probability-flow ODE

$x_q=(u,v)$4

where $x_q=(u,v)$5 is the teacher score network and $x_q=(u,v)$6 is the noise schedule (Gu et al., 13 May 2026). The distilled model is trained to approximate the exact ODE solution mapping

$x_q=(u,v)$7

so that $x_q=(u,v)$8.

The paper’s central claim is that distilling full ODE transitions rather than only endpoints preserves the desirable property that increasing the number of solver steps at test time monotonically improves approximation of the ODE trajectory (Gu et al., 13 May 2026). Test-time sampling can therefore use an Euler scheme with $x_q=(u,v)$9 steps:

$[0,H/8]\times[0,W/8]$0

The framework is presented as the first any-step video diffusion distillation method built around this two-time transition formulation.

6. Distillation, backward simulation, and empirical results in video generation

After an initial forward flow-map training stage, the diffusion AnyFlow refines $[0,H/8]\times[0,W/8]$1 via an on-policy Distribution-Matching Distillation loss (Gu et al., 13 May 2026). For a sampled interval $[0,H/8]\times[0,W/8]$2 derived from a target budget $[0,H/8]\times[0,W/8]$3, the model produces a three-segment rollout:

$[0,H/8]\times[0,W/8]$4

It then re-noises $[0,H/8]\times[0,W/8]$5 back to $[0,H/8]\times[0,W/8]$6 via the known forward process and minimizes the KL divergence between teacher and student predicted distributions at $[0,H/8]\times[0,W/8]$7:

$[0,H/8]\times[0,W/8]$8

The expectation is over the target step budget $[0,H/8]\times[0,W/8]$9, the sampled split, initial noise, and prompt context (Gu et al., 13 May 2026).

The paper further introduces Flow Map Backward Simulation, which decomposes a full Euler rollout into shortcut flow-map transitions (Gu et al., 13 May 2026). In practice, shortcut calls $s$00 replace long multistep Euler rollouts and thereby reduce training cost. The method is motivated by two failure modes: discretization error in few-step sampling and exposure bias in causal generation. By training over all intervals $s$01 and using on-policy rollouts, AnyFlow is reported to correct coarse-step ODE errors and reduce drift in causal models.

The implementation spans both bidirectional and causal architectures (Gu et al., 13 May 2026). The bidirectional variant uses a standard UNet backbone, Wan2.1, distilled via the AnyFlow pipeline. The causal variant uses FAR context compression with three “full-token” chunks at patch size 2, the remainder at patch size 4, and a first chunk of size 1 for precise first-frame conditioning. KV caches are reused between flow-map calls to speed up backward simulation. Reported parameter scales are 1.3 B for Wan2.1-1.3B and 14 B for Wan2.1-14B.

The paper also specifies forward training tricks and hyperparameters (Gu et al., 13 May 2026). These include interpolated timestep conditioning with $s$02 and $s$03, guidance-fused training, adaptive loss reweighting anchored at boundary cases $s$04, and a time sampler uniform over $s$05 with an optional timestep reweight function $s$06. Typical hyperparameters are AdamW, learning rate $s$07 in stage 1 and $s$08 in stage 2, weight decay $s$09, batch size $s$10 GPUs times $s$11 samples each, stage 1 of approximately $s$12 K iterations, stage 2 of approximately $s$13 K iterations, and guidance scale matched to the pretrained teacher.

Selected VBench totals are reported for the 14 B models (Gu et al., 13 May 2026). For bidirectional text-to-video, rCM at 4 NFE scores $s$14, AnyFlow at 4 NFE scores $s$15, and AnyFlow at 32 NFE scores $s$16. For causal text-to-video, Krea-Realtime at 4 NFE scores $s$17, AnyFlow-FAR at 4 NFE scores $s$18, and AnyFlow-FAR at 32 NFE scores $s$19. For image-to-video, Wan2.1-I2V at 100 NFE scores $s$20, while AnyFlow-FAR at 4 NFE scores $s$21. The paper interprets these results as evidence that AnyFlow matches or surpasses consistency-based counterparts in the few-step regime while scaling with sampling step budgets.

7. Relationship, distinctions, and limitations

The two AnyFlow systems are unrelated at the task level. One is an optical-flow estimator for dense motion between image pairs; the other is a video diffusion distillation framework for generative sampling (Jung et al., 2023, Gu et al., 13 May 2026). The shared name can therefore be misleading if treated as designating a single method family.

Their methodological parallel lies in representing outputs through continuous mappings rather than fixed discrete endpoints. In the optical-flow case, the continuity is spatial and coordinate-based, allowing arbitrary-scale querying of flow fields (Jung et al., 2023). In the diffusion case, the continuity is temporal and interval-based, allowing arbitrary transition learning between times $s$22 and $s$23 and supporting any-step inference (Gu et al., 13 May 2026). This suggests that “AnyFlow” functions as a label for models that seek robustness under variable resolution or variable solver budget by learning over continuous domains.

The limitations discussed in the two papers are likewise domain-specific. For optical flow, the iterative GRU backbone remains the dominant compute cost; the region encoding and dynamic-radius MLPs introduce hyperparameters such as the initial $s$24 and auxiliary patch size; and the authors identify unsupervised or self-supervised photometric losses and integration with Transformer-style global matching as future directions (Jung et al., 2023). For video diffusion, the paper frames its contribution against the limitations of consistency distillation and positions flow-map learning as a remedy for degraded performance at higher sampling budgets, but its concrete claims are primarily about on-policy distillation, ODE-consistent scaling, and broad applicability across bidirectional and causal architectures rather than about unresolved weaknesses (Gu et al., 13 May 2026).

Taken together, the term “AnyFlow” currently refers to two separate contributions that use continuity as a design principle to overcome rigidity in standard formulations: fixed-grid prediction in optical flow, and fixed-step or endpoint-only distillation in video diffusion.