RelayFormer: Unified Visual Manipulation Localization

• The paper introduces RelayFormer, a unified framework that employs local units and relay tokens to capture fine-grained forensic cues in dense mask prediction for visual manipulation localization.
• The architecture integrates local-aware attention and relay-based global attention to balance resolution agnosticism and computational efficiency, demonstrating linear scalability with respect to unit count.
• RelayFormer adapts pretrained Transformer backbones with minimal additional parameters, achieving state-of-the-art performance on benchmark image and video forensic tasks.

Searching arXiv for the exact papers and closely related context. arXiv_search query: "RelayFormer visual manipulation localization (Huang et al., 13 Aug 2025)" RelayFormer most directly denotes a unified and modular architecture for visual manipulation localization across images and videos, introduced in “RelayFormer: A Unified Local-Global Attention Framework for Scalable Image and Video Manipulation Localization” (Huang et al., 13 Aug 2025). In that work, the task is dense mask prediction of tampered regions in visual content, with a particular emphasis on subtle, sparse, high-frequency, and semantically simple forensic cues. The name should not be conflated with “Learned Relay Representations for Forward-Thinking Discrete Diffusion Models,” whose method is called “Relay,” not “RelayFormer,” and which addresses masked diffusion models for discrete sequence generation rather than visual forensics (Rozonoyer et al., 21 May 2026).

1. Nomenclature and problem formulation

RelayFormer is defined for visual manipulation localization (VML): given an image or a video, predict a dense mask that localizes tampered regions. In the image setting, the target manipulations include splicing, copy-move, retouching, and inpainting. In the video setting, the paper focuses mainly on video inpainting localization, where manipulated regions must be localized frame by frame while also exploiting temporal consistency (Huang et al., 13 Aug 2025).

The motivating observation is that VML differs from conventional semantic segmentation. The discriminative evidence often resides in boundary inconsistencies, local texture disruptions, or forensic traces rather than in rich object-level semantics. Because these cues are fine-grained, aggressive resizing can destroy them. RelayFormer is therefore organized around three stated objectives: preserving fine local forensic evidence, supporting arbitrary input resolution and video length, scaling better than dense global attention, and remaining compatible with pretrained Transformer backbones.

A central claim of the work is that earlier VML systems are often modality-specific, fixed-resolution, or computationally burdened by dense global self-attention. RelayFormer addresses these issues through a single architecture that uses local processing units, relay-mediated global communication, and a lightweight query-based decoder. This suggests that the framework is intended less as a new standalone backbone than as a reusable VML adaptation layer for pretrained Transformer systems.

2. Local units and the relay-token abstraction

RelayFormer partitions an input frame of resolution $H \times W$ into overlapping local units $U$ of size $P \times P$, with padding used when necessary so the whole image is covered. The number of units per frame is

$m = \lceil H/P \rceil \times \lceil W/P \rceil .$

For video, the same construction is extended to spatial-temporal units. The implementation details reported in the paper use image unit size $512\times512$, video unit size $224\times224$, and clip length $4$ (Huang et al., 13 Aug 2025).

This unit-based construction is the first scalability mechanism. Rather than resizing every sample to a common large resolution or applying full-image attention, the model processes modular units at native detail. The paper attributes three benefits to this choice: resolution agnosticism, less redundant computation than padding all inputs to a fixed maximum size, and preservation of local information.

Within each unit $U_i$, RelayFormer maintains patch-token features

$X_i \in \mathbb{R}^{N \times d},$

and appends a small set of learnable Global Relay Tokens (GRTs)

$T_i \in \mathbb{R}^{n \times d},$

where $U$0 is the number of GRTs per unit. The role of the GRTs is not to replace local patch features but to absorb unit-level manipulation cues and carry them into a global communication stage. The term “relay” is literal: information is transmitted across units indirectly through these compact token sets rather than through dense all-token attention.

The same abstraction is used for both images and videos. For images, units are spatial patches; for videos, units extend naturally over space and time. The paper explicitly states that GLoRA “naturally extends to temporal modeling,” which is the basis for the architecture’s cross-modal reuse (Huang et al., 13 Aug 2025).

3. Global-Local Relay Attention (GLoRA)

The core mechanism of RelayFormer is Global-Local Relay Attention (GLoRA), which consists of two stages: Local-aware Attention and Relay-based Global Attention. GLoRA is inserted after standard Transformer blocks and is designed to reuse backbone weights while introducing lightweight LoRA-based adaptation to switch between local and global modes (Huang et al., 13 Aug 2025).

In the local stage, self-attention is applied within each unit over the concatenated sequence of relay tokens and patch tokens:

$U$1

where $U$2 is the layer index and $U$3 denotes concatenation. This means that patch tokens interact with other patch tokens in the same unit, while GRTs attend to local patch content and absorb local forensic evidence.

After local aggregation, all GRTs from all units are concatenated into

$U$4

and global communication is performed only over this relay-token sequence. To encode token identity together with spatial and temporal positions, the framework applies 4D Rotary Positional Embedding:

$U$5

The updated relay tokens are then redistributed back to their corresponding units for the next layer, where subsequent local attention diffuses the injected global context back into patch tokens. The resulting communication pattern is therefore: local unit $U$6 local GRTs, global mixing among all GRTs, updated GRTs returned to units, and next-layer local attention spreads global context back into local tokens.

The paper gives the overall time complexity of GLoRA as

$U$7

with $U$8 denoting the number of units, $U$9 the number of patch tokens per unit, and $P \times P$0 the number of GRTs per unit. The first term corresponds to local attention within units; the second corresponds to global attention among relay tokens. Because $P \times P$1, the global communication channel is much smaller than the full patch-token sequence. In practice, the paper reports that $P \times P$2 is sufficient to achieve the best accuracy in ablations (Huang et al., 13 Aug 2025).

The paper does not provide the internal multi-head attention equations for $P \times P$3 or $P \times P$4, nor the explicit LoRA parameterization equations inside GLoRA. It also does not formalize the token redistribution rule after $P \times P$5. Those omissions are implementation-relevant because they bound the level of exact reproducibility available from the main text alone.

4. Backbone adaptation and mask decoding

RelayFormer is not presented as a new backbone trained from scratch. It is a modular adapter framework for pretrained Transformer-based backbones, demonstrated with ViT and SegFormer, yielding Relay-ViT and Relay-Seg respectively. The paper states that GLoRA is inserted after standard Transformer blocks and uses two sets of LoRA adapters to enable switching between local and global modes with minimal parameter overhead. In the appendix, it further states that all Transformer blocks in the backbone are replaced with GLoRA modules (Huang et al., 13 Aug 2025).

The reported additional parameter counts are small relative to the base models: Relay-ViT is $P \times P$6M parameters, and Relay-Seg is $P \times P$7M. The intended effect is to preserve the original attention structure and pretrained weights while introducing lightweight task-specific adaptation. This is particularly relevant because VML datasets are much smaller than large-scale pretraining corpora, so preserving pretrained representations is beneficial.

For mask prediction, RelayFormer uses a lightweight query-based Transformer decoder inspired by Mask2Former-style decoding. Given the reassembled feature map

$P \times P$8

it is projected to a lower-dimensional representation

$P \times P$9

A fixed set of learnable queries is then used:

$m = \lceil H/P \rceil \times \lceil W/P \rceil .$0

At decoder layer $m = \lceil H/P \rceil \times \lceil W/P \rceil .$1, the queries are updated by cross-attention and then self-attention with rotary positional encoding:

$m = \lceil H/P \rceil \times \lceil W/P \rceil .$2

$m = \lceil H/P \rceil \times \lceil W/P \rceil .$3

The appendix specifies decoder depth $m = \lceil H/P \rceil \times \lceil W/P \rceil .$4 and number of learnable queries $m = \lceil H/P \rceil \times \lceil W/P \rceil .$5. A gating MLP assigns weights to each query to modulate its contribution to the final output. The paper does not provide the exact final mask-generation equation or the explicit gating MLP formula, and it does not describe Hungarian matching, bipartite assignment, or DETR-style set loss.

For video, RelayFormer adopts one-shot inference: a shared set of queries from the first frame is used to decode masks for the entire video clip via dot-product attention. The stated motivation is that VML has relatively low semantic complexity and fairly consistent feature structure across adjacent frames. The paper reports that this significantly reduces decoder GFLOPs while maintaining accuracy.

5. Optimization, supervision, and evaluation protocol

RelayFormer is trained with a combination of binary cross-entropy and edge loss:

$m = \lceil H/P \rceil \times \lceil W/P \rceil .$6

where $m = \lceil H/P \rceil \times \lceil W/P \rceil .$7 is the predicted mask, $m = \lceil H/P \rceil \times \lceil W/P \rceil .$8 is the ground-truth mask, and $m = \lceil H/P \rceil \times \lceil W/P \rceil .$9 is the edge mask. The edge term is itself BCE over the edge region:

$512\times512$0

The appendix sets $512\times512$1 (Huang et al., 13 Aug 2025).

For images, the paper follows the standard cross-dataset protocol: training on CASIA v2.0 and testing on Coverage, Columbia, NIST16, CASIA v1.0, IMD2020. For videos, training data is created from DAVIS2016 using two video inpainting models, OP and VI, and testing is performed cross-dataset on MOSE validation clips edited by E2FGVI, FuseFormer, and STTN. The evaluation metrics are F1 at threshold $512\times512$2 and IoU.

Using IMDLBench, the optimization configuration is reported as follows: AdamW, 200 epochs, base learning rate $512\times512$3, cosine decay, weight decay $512\times512$4, warmup during the first 2 epochs, minimum learning rate $512\times512$5, AMP mixed precision, 4 RTX 3090 GPUs, per-GPU batch size 2, gradient accumulation 8 steps, and effective batch size 64. A stabilization step is used in the first epoch: pretrained parameters are frozen and only newly introduced parameters are trained.

The appendix states that augmentation follows IML-ViT and includes rescaling, horizontal flipping, Gaussian blur, random rotation, and synthetic copy-move and inpainting of rectangular regions within the same image. No test-time augmentation or post-processing is used.

The paper also notes a tension between the conceptual emphasis on avoiding interpolation and an implementation detail: for images larger than $512\times512$6, the longer side is resized to 1024 while preserving aspect ratio, then zero-padded to $512\times512$7. This does not negate the unit-based design, but it constrains the literal extent to which the implementation is entirely interpolation-free.

6. Empirical behavior, scaling properties, and relation to adjacent “relay” work

On image manipulation localization under the MVSS protocol, RelayFormer reports the best average F1 = 0.543, achieved by both Relay-ViT and Relay-Seg. Per-dataset highlights reported in the paper include Relay-Seg obtaining the best result on Coverage with 0.569, Relay-ViT obtaining the best on NIST16 with 0.299, and Relay-Seg obtaining the best on IMD2020 with 0.357. On video manipulation localization on MOSE, Relay-Seg leads on E2FGVI with 0.561 / 0.698 (IoU/F1), while for STTN, Relay-ViT reaches 0.549 / 0.684 and Relay-Seg reaches 0.534 / 0.674 (Huang et al., 13 Aug 2025).

The ablations attribute the gains to both relay tokens and decoder design. The reported image ablation values are: no GRTs plus MLP decoder, average 0.454; $512\times512$8 plus MLP decoder, 0.521; $512\times512$9 plus proposed decoder, 0.532; $224\times224$0 plus proposed decoder, 0.543; $224\times224$1 plus proposed decoder, 0.524. For the spatial and temporal branches on MOSE, the reported F1 values are 0.6124 with no spatial and no temporal branch, 0.6745 with spatial only, and 0.6877 with spatial plus temporal. Appendix results on layer replacement strategy show 73.24 for sparse insertion, 74.21 for latter-half insertion, and 75.50 for full replacement. The paper also reports an interpolation effect on IMD2020: F1 = 0.453 at full resolution with no resize versus F1 = 0.350 when resized to $224\times224$2.

The system-level complexity numbers are also reported explicitly. For Relay-ViT, GFLOPs are 119.18 at $224\times224$3, 238.20 at $224\times224$4, and 476.12 at $224\times224$5. For Relay-Seg, they are 52.71, 105.41, and 210.83 respectively. The paper characterizes this as approximately linear scaling in the number of units at the system level and emphasizes that units can be grouped by count and processed with strong intra-group parallelism.

Several practical limitations are either stated or can be inferred from the reported setup. The paper is relatively light on explicit failure-case discussion. A plausible implication is that performance remains sensitive to the loss of high-frequency evidence, since resizing degrades IMD2020 performance substantially. Another plausible implication is that extreme long-range temporal behavior is not fully characterized, because the reported video clip length is $224\times224$6. In addition, while GLoRA is more scalable than full attention, its global relay attention still contains a quadratic term in the total number of relay tokens, $224\times224$7, so the improvement is one of compression rather than unbounded scalability.

The “relay” terminology in RelayFormer should also be distinguished from the unrelated use of “Relay” in discrete diffusion modeling. “Learned Relay Representations for Forward-Thinking Discrete Diffusion Models” introduces a differentiable per-token channel that propagates latent state across denoising rounds in masked diffusion models, with recurrence over $224\times224$8 and training via truncated BPTT (Rozonoyer et al., 21 May 2026). RelayFormer, by contrast, uses relay tokens to mediate sparse global communication across spatial-temporal units for VML. The shared vocabulary reflects a common intuition of passing intermediate information forward, but the problem domains, architectural objects, and training dynamics are distinct.