Residual Features Refilling (RFR)

• Residual Features Refilling is a dual-path approach that restores spatial detail by supplementing compressed features with residual signals.
• The technique enhances high-dimensional visual processing via high-resolution branches and explicit feature alignment through upsampling and shuffling operations.
• Empirical results show that integrating RFR with pixel-unshuffle modules improves segmentation accuracy and video codec performance over baseline methods.

Residual Features Refilling (RFR) is a technique designed to address the resolution loss caused by compression operations in neural architectures that process high-dimensional visual signals, particularly in multi-modal LLMs (MLLMs) and hybrid video codecs. The core principle behind RFR is to restore fine-grained spatial information by explicitly retaining uncompressed high-resolution features and subsequently refilling them with residuals derived from network-internal transformations. This dual-pathway paradigm enables the restoration of spatial precision critical for downstream tasks such as segmentation or artifact-free video reconstruction, while preserving the semantic richness learned by the backbone network (Zhang et al., 19 Mar 2026, Jia et al., 2019).

1. Rationale and Conceptual Basis

RFR arises from the observation that compression and token reduction operations (such as pixel-shuffle and MLP compression) in standard MLLMs degrade the spatial fidelity necessary for dense prediction tasks. In "Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token" (Zhang et al., 19 Mar 2026), this issue is addressed via RFR, which refills the lost spatial detail in high-resolution feature maps by injecting residuals computed between LLM-processed compressed tokens and their deeply compressed encodings. The approach is closely related to techniques in the video coding domain, where residual signals, typically discarded after frame reconstruction, are leveraged as first-class inputs in convolutional models to enhance in-loop filtering (Jia et al., 2019).

The guiding philosophy is to treat residual information not as noise, but as a carrier of high-frequency detail and geometric cues (e.g., partition boundaries in codecs, or spatial detail in visual encoder outputs) that materially benefit the target reconstruction or segmentation operation.

2. Feature Retention and Residual Computation

In the MLLM segmentation context, consider an input image $x$ processed by a visual encoder $\mathcal{E}$ to yield pre-shuffled features $F_{V_0}\in\mathbb{R}^{N_0\times d_0}$, where $N_0=H\cdot W$. The network applies a pixel-shuffle and MLP stack with downsampling factor $\alpha$ to obtain compressed features $F_{V_1}\in\mathbb{R}^{N_1\times d}$, $N_1=N_0/\alpha$, $d=\alpha d_0$.

RFR intervenes by creating a high-resolution branch:

• Replicate each $F_{V_0}$ token $\alpha$ times, creating $\mathcal{E}$0.
• Apply the compression MLP independently to generate $\mathcal{E}$1 (high-quality, uncompressed features).

After MLLM backbone and LLM processing (including the [SEG] token), the LLM-refined compressed image features $\mathcal{E}$2 are available. The residual is defined as: $\mathcal{E}$3 which quantifies the increment due to deep network processing, at reduced resolution.

This residual $\mathcal{E}$4 is upsampled, e.g., by nearest-neighbor interpolation, to match the high-resolution grid: $\mathcal{E}$5 The refilled feature map is

$\mathcal{E}$6

This procedure restores details removed during compression, thereby combining the original spatial precision of $\mathcal{E}$7 with the semantic refinement encoded in $\mathcal{E}$8.

3. Residual Features Amplifier: Pixel-Shuffle/Unshuffle Enhancement

To further exploit residual information, RFR incorporates a Residual Features Amplifier (RFA) that interleaves pixel-shuffle and pixel-unshuffle operations with learnable MLPs:

• Pixel-shuffle $\mathcal{E}$9 maps $F_{V_0}\in\mathbb{R}^{N_0\times d_0}$0.
• Pixel-unshuffle $F_{V_0}\in\mathbb{R}^{N_0\times d_0}$1 maps $F_{V_0}\in\mathbb{R}^{N_0\times d_0}$2.

Two pixel-unshuffle+MLP blocks, $F_{V_0}\in\mathbb{R}^{N_0\times d_0}$3 and $F_{V_0}\in\mathbb{R}^{N_0\times d_0}$4, process both compressed ($F_{V_0}\in\mathbb{R}^{N_0\times d_0}$5) and LLM-processed ($F_{V_0}\in\mathbb{R}^{N_0\times d_0}$6) features to yield $F_{V_0}\in\mathbb{R}^{N_0\times d_0}$7 and $F_{V_0}\in\mathbb{R}^{N_0\times d_0}$8 in the high-resolution domain. The amplified residual

$F_{V_0}\in\mathbb{R}^{N_0\times d_0}$9

is fused with the unshuffled HQ features $N_0=H\cdot W$0, yielding the enhanced refilled map: $N_0=H\cdot W$1 This stack increases the expressive power of RFR, enabling finer discrimination between regions with residual semantic detail and relatively stable reconstructions.

4. Integration with Token-Level Attention and Decoding

RFR is accompanied by a custom dual-pathway attention mask. Specifically, bidirectional attention is enabled among image tokens and between image tokens and the [SEG] segmentation token, while preserving causal constraints for textual tokens. The segmentation mask is derived by

$N_0=H\cdot W$2

where $N_0=H\cdot W$3 is the embedding for the [SEG] token. This architecture allows efficient, decoder-free mask generation directly from the segment token, in contrast to conventional approaches relying on specialist decoders, thereby simplifying the pipeline while leveraging the full benefit of residual refilling (Zhang et al., 19 Mar 2026).

5. Empirical Impact and Ablation Findings

Table: Core cIoU (%) Results for RefCOCO Segmentation, as Reported in (Zhang et al., 19 Mar 2026)

Method	val	testA	testB
Baseline (no HQ/RFR)	76.6	78.3	73.6
+ HQ features only	78.7	81.1	75.2
+ HQ + RFR	79.0	81.3	76.3
+ HQ + RFR + Pixel-Unshuffle	79.8	82.0	77.1

Ablation studies show that RFR consistently yields up to ≈1% cIoU improvement over the HQ-only branch, and that the combination with pixel-unshuffle modules further amplifies this gain. Across benchmarks including RefCOCO+, RefCOCOg, ReasonSeg, and open-vocabulary segmentation, methods incorporating RFR match or outperform architecturally more complex specialized decoders.

6. Analogy and Lineage: Residual Refilling in Video Coding

Direct analogues of RFR appear in video compression research, such as in Residual-Reconstruction-based Convolutional Neural Networks (RRNet) for in-loop filtering (Jia et al., 2019). In this context, the prediction residual $N_0=H\cdot W$4—extracted post entropy decoding, inverse quantization, and inverse DCT/DST—is not discarded after forming the reconstructed frame $N_0=H\cdot W$5: $N_0=H\cdot W$6 Instead, $N_0=H\cdot W$7 is normalized and fused with $N_0=H\cdot W$8 in a dual-branch CNN designed to restore texture and edge information lost due to quantization. The concatenated feature maps are merged via a $N_0=H\cdot W$9 fusion convolution. This architecture enables the model to “refill” block boundaries and textured regions in reconstructed images, resulting in up to $\alpha$0 BD-rate savings on high-complexity content classes compared to HEVC baselines (Jia et al., 2019).

In both modalities, the refilled residual pathway specializes on regions with high error or quantization artifacts, substantially boosting localized restoration while maintaining global consistency.

7. Significance and Research Trajectory

RFR establishes a general framework for leveraging internally available residual signals—whether from visual encoders in MLLMs or from bitstream artifacts in video codecs—for spatial detail recovery. The technique is characterized by:

• Direct feed-in of high-resolution branches alongside compressed or semantically processed features.
• Construction and spatial alignment of residuals through explicit, mathematically well-defined operations (upsampling, shuffling, unshuffling).
• Demonstrable improvement on pixel-level variants of vision-language tasks and established video compression benchmarks.
• Compatibility with end-to-end differentiable training, enabling gradient propagation that specializes the residual branch toward difficult regions (e.g., textured or strongly quantified areas).

A plausible implication is that RFR and analogous dual-branch refilling schemes will continue to inform hybrid architectures for tasks where spatial precision and deep semantic representation must coexist, pointing toward future model architectures that sidestep purely decoder-based approaches in favor of pipeline-unified refilling and feature fusion.

References:

(Zhang et al., 19 Mar 2026) "Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token" (Jia et al., 2019) "Residual-Guided In-Loop Filter Using Convolution Neural Network"