HRSeg: High-Res Reasoning Segmentation

• HRSeg is a high-resolution, reasoning-aware segmentation framework that combines global and local features via HRP and HRE modules to interpret complex visual and textual cues.
• It overcomes the limitations of moderate-resolution encoders by avoiding high-res positional-embedding interpolation, using efficient crop-based processing instead.
• Empirical results show significant IoU improvements on ReasonSeg, LLM-Seg40K, and RefCOCO benchmarks, validating its effectiveness in reasoning and referring segmentation.

HRSeg is a high-resolution, reasoning-aware segmentation framework built on top of multimodal LLMs (MLLMs) and SAM for the reasoning segmentation task, in which an image and an implicit natural-language instruction must be mapped to one or more segmentation masks (Lin et al., 17 Jul 2025). It is designed to address the low-perceptual-resolution limitations of standard visual encoders without relying on direct high-resolution positional-embedding interpolation. Its two defining modules are High-Resolution Perception (HRP), which integrates global and cropped local features, and High-Resolution Enhancement (HRE), which refines mask features using high-resolution contextual information. In the reported experiments, the framework improves reasoning segmentation on ReasonSeg and LLM-Seg40K and also generalizes to referring segmentation benchmarks (Lin et al., 17 Jul 2025).

1. Task formulation and motivation

Reasoning segmentation requires interpreting implicit textual instructions that may depend on contextual cues or open-world knowledge, such as segmenting “the person who looks the most worried” or “the chair that is closest to the window” (Lin et al., 17 Jul 2025). This distinguishes it from semantic segmentation, where targets are explicit categories, and from referring segmentation, where expressions are usually shorter and grounded in direct local cues. The central difficulty is therefore not only pixel-level delineation, but the joint resolution of instruction understanding, target disambiguation, and fine-grained visual grounding.

Two pre-existing paradigms are identified. The first is end-to-end LLM+SAM training, exemplified by LISA, PixelLM, and LLaVASeg, where a <SEG> token is introduced and a SAM-style decoder is fine-tuned. This setting is resource-intensive because it requires fine-tuning SAM’s decoder and substantial training data. The second is the two-stage reasoning-and-selection strategy of LLM-Seg, where a frozen SAM generates mask proposals and an MLLM selects the correct proposal. That approach is cheaper to train, but its visual encoder typically operates at moderate resolution, such as $518 \times 518$ for DINOv2, which weakens discrimination of small objects and subtle local cues (Lin et al., 17 Jul 2025).

A further motivation concerns naive high-resolution scaling. Interpolating positional embeddings of a ViT-based visual encoder to higher resolutions yields only modest improvements, and performance may even decline as resolution increases. At the same time, the complexity of a ViT remains quadratic in the number of tokens, so increasing spatial resolution substantially increases FLOPs and memory. HRSeg is positioned as an alternative that exploits high-resolution information efficiently while preserving the pre-training regime of the base visual encoder (Lin et al., 17 Jul 2025).

2. System architecture

HRSeg is organized as a two-branch framework consisting of a reasoning branch and a segmentation branch (Lin et al., 17 Jul 2025). The reasoning branch uses an MLLM, such as LLaVA-7B-v1.1 or LLaVA-1.6, with a vision encoder, a visual projector, and a LLM. A special token <SEG> is added to the vocabulary, and the embedding associated with this token is used as the text-conditioned segmentation cue: $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$ Here, $\tilde{Y}_{\text{seg}}$ serves as the reasoning-conditioned feature used for mask selection.

The segmentation branch begins with a frozen SAM, which receives the image and produces a set of candidate masks from a grid of point prompts, for example a $32 \times 32$ grid: $$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$ In parallel, HRSeg adds a separate frozen DINOv2-ViT-L encoder $f_v$, pre-trained at $518 \times 518$. The image is represented as a global low-resolution view $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$ and a local high-resolution view $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$, where the default magnification is $N=2$. The high-resolution view is processed by sliding-window cropping with windows of size $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$0, so the encoder can be reused without changing its architecture or positional embeddings: $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$1

HRP takes the candidate masks and the global and local image features and produces hybrid image features $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$2 together with per-mask features $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$3: $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$4 HRE then refines these mask features: $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$5

The final selection module injects textual semantics into the refined mask features through self-attention and cross-attention with $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$6. It computes a text-similarity score

$$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$7

and an IoP score $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$8, where IoP estimates how much of a proposal overlaps the target object. The selected output mask or masks are denoted

$$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$9

This architecture preserves SAM as a frozen proposal generator while transferring fine-grained disambiguation to HRP, HRE, and the text-conditioned selection mechanism (Lin et al., 17 Jul 2025).

3. High-Resolution Perception and High-Resolution Enhancement

HRP is the principal mechanism by which HRSeg recovers high-resolution detail (Lin et al., 17 Jul 2025). Global features $\tilde{Y}_{\text{seg}}$0 and local cropped features $\tilde{Y}_{\text{seg}}$1 are reshaped into feature maps, with $\tilde{Y}_{\text{seg}}$2 mapped to $\tilde{Y}_{\text{seg}}$3 and $\tilde{Y}_{\text{seg}}$4 mapped to $\tilde{Y}_{\text{seg}}$5, where $\tilde{Y}_{\text{seg}}$6. A region extraction operator selects, for each global token, a corresponding $\tilde{Y}_{\text{seg}}$7 local patch: $\tilde{Y}_{\text{seg}}$8 This produces region features

$\tilde{Y}_{\text{seg}}$9

A region-attention operation then enriches each global token using its corresponding local patch: $32 \times 32$0 The resulting region-refined feature is combined with the original global feature by a weighted sum: $32 \times 32$1 The reported ablation identifies $32 \times 32$2 as the optimal setting. This balance indicates that global semantics remain dominant while local detail supplies targeted refinement rather than a wholesale replacement of the global representation.

HRP also converts binary mask proposals into object-level features by masked average pooling: $32 \times 32$3 Each proposal is therefore represented by a feature vector that reflects high-resolution content within the mask. A plausible implication is that HRP turns SAM’s mask proposals into semantically comparable entities rather than treating them as purely geometric regions.

HRE addresses the complementary problem that in-mask pooling alone may miss context outside the mask boundary. It applies stacked cross-attention layers, with depth $32 \times 32$4 in the experiments, between mask features and the hybrid image representation: $32 \times 32$5 followed by residual addition and normalization: $32 \times 32$6 Through this design, each mask feature can attend to the entire hybrid image representation and aggregate context relevant to relational or comparative instructions. This suggests that HRE is especially important when the decisive evidence for a mask lies outside the candidate region itself, such as in “the person holding a cup” or “the second person from the left” (Lin et al., 17 Jul 2025).

4. Training protocol and implementation

HRSeg is trained on a mixture of semantic segmentation, referring segmentation, and reasoning segmentation datasets (Lin et al., 17 Jul 2025). The semantic segmentation component includes COCO-Stuff, ADE20K, PACO-LVIS, PartImageNet, and PASCAL-Part. The referring segmentation component includes RefClef, RefCOCO, RefCOCO+, and RefCOCOg. The reasoning segmentation component includes ReasonSeg and LLM-Seg40K. This training mixture is intended to preserve general grounding ability while introducing more complex instruction following.

The total loss is

$32 \times 32$7

with both weights set to $32 \times 32$8. The text generation term $32 \times 32$9 is a standard autoregressive cross-entropy objective over MLLM outputs. The mask-selection term is

$$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$0

Its first component matches the similarity distribution against the IoU distribution: $$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$1 while the second performs IoP regression: $$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$2 with

$$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$3

This decomposition is notable because it separately models semantic relevance and proposal quality.

The reported implementation uses LLaVA-7B-v1.1 with LoRA fine-tuning, together with variants based on LLaVA-1.6. HRP employs a frozen DINOv2-ViT-L encoder pre-trained at $$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$4. Training is performed with DeepSpeed on $$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$5 NVIDIA 3090 GPUs, with batch size $$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$6 per GPU, gradient accumulation $$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$7, AdamW, and WarmupDecayLR with initial learning rate $$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$8. The default high-resolution setting uses magnification $$\mathbf{M} = \{m_1, m_2, \ldots, m_K\} \in \mathbb{R}^{K \times D}.$$9, corresponding to $f_v$0 images, while the crop window remains fixed at $f_v$1. HRSeg does not interpolate positional embeddings of the ViT; instead, it keeps the ViT at the pre-training resolution and simulates higher resolution by cropping (Lin et al., 17 Jul 2025).

5. Empirical results and ablation findings

The main evaluations are reported on ReasonSeg, LLM-Seg40K, and the RefCOCO series, using gIoU and cIoU for reasoning segmentation and cIoU for referring segmentation (Lin et al., 17 Jul 2025). The clearest quantitative pattern is that HRSeg improves both average IoU and cumulative IoU over LLM-Seg, with further gains when paired with a stronger MLLM.

Benchmark and setting	Baseline	HRSeg
ReasonSeg val, no fine-tuning	LLM-Seg: 47.4 gIoU / 35.4 cIoU	50.1 gIoU / 47.1 cIoU
ReasonSeg val, fine-tuning	LLM-Seg: 52.3 / 47.5	57.4 / 54.7
LLM-Seg40K val, no fine-tuning	LLM-Seg: 36.0 / 39.4	44.3 / 44.4
LLM-Seg40K val, fine-tuning	LLM-Seg: 45.5 / 54.2	58.7 / 58.0

On ReasonSeg, the fine-tuned HRSeg model improves over fine-tuned LLM-Seg by $f_v$2 gIoU and $f_v$3 cIoU on the validation set. Without fine-tuning on ReasonSeg, HRSeg improves over LLM-Seg on the test set by $f_v$4 gIoU and $f_v$5 cIoU. The variant using LLaVA-1.6 reaches $f_v$6 gIoU and $f_v$7 cIoU on ReasonSeg validation after fine-tuning, and $f_v$8 gIoU and $f_v$9 cIoU on the test set overall. The reported gains are larger on long-query instructions: the gain in gIoU on long queries is $518 \times 518$0, which is $518 \times 518$1 higher than the gain on short queries at $518 \times 518$2. This suggests that improved high-resolution perception primarily benefits cases where instruction resolution and visual discrimination are jointly difficult (Lin et al., 17 Jul 2025).

On LLM-Seg40K, the validation score increases from $518 \times 518$3 gIoU and $518 \times 518$4 cIoU for fine-tuned LLM-Seg to $518 \times 518$5 gIoU and $518 \times 518$6 cIoU for fine-tuned HRSeg. The paper characterizes the $518 \times 518$7 gIoU improvement as particularly substantial across diverse photographic and egocentric images.

HRSeg also generalizes to referring segmentation. With LLaVA-1.6, it reaches cIoU values of $518 \times 518$8 on RefCOCO val, $518 \times 518$9 on RefCOCO+ val, and $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$0 on RefCOCOg val(U), achieving or surpassing the reported state of the art.

The ablation studies isolate the effects of HRP and HRE. On ReasonSeg validation, the fine-tuned LLM-Seg baseline gives $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$1 gIoU and $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$2 cIoU; adding HRP alone increases this to $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$3 and $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$4, and adding both HRP and HRE increases it to $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$5 and $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$6. On LLM-Seg40K validation, the same progression is $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$7. These results indicate that HRP supplies most of the fine-grained gain, while HRE provides an additional context-sensitive improvement.

The magnification-factor ablation reports a consistent increase with higher $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$8: on ReasonSeg validation, the sequence is $I_g \in \mathbb{R}^{H_v \times W_v \times 3}$9 without HRSeg high-resolution processing, $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$0 for $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$1, $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$2 for $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$3, and $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$4 for $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$5. The authors nevertheless select $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$6 as the default because it gives a favorable accuracy-efficiency trade-off.

The mask-selection ablation shows that combining semantic similarity with proposal quality is preferable to using either alone. On ReasonSeg validation, top-1 selection by $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$7 gives $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$8, top-1 by $$I_l \in \mathbb{R}^{(N H_v) \times (N W_v) \times 3}$$9 gives $N=2$0, selecting all masks with $N=2$1 gives $N=2$2 at $N=2$3, top-$N=2$4 by $N=2$5 gives $N=2$6, and the intersection strategy top-$N=2$7 $N=2$8 gives $N=2$9, with the best threshold at $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$00, where the score becomes $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$01 (Lin et al., 17 Jul 2025).

6. Computational profile, limitations, and related usages of the term

HRSeg’s computational rationale is explicit: a naive high-resolution ViT approach based on positional-embedding interpolation scales quadratically with resolution, whereas HRSeg keeps each ViT forward pass at the original $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$02 pre-training resolution and applies it to a moderate number of crops (Lin et al., 17 Jul 2025). The complexity therefore grows approximately linearly in the number of crops rather than quadratically in the full-image token count. Region attention is also localized, because each global token attends only to its corresponding local high-resolution patch rather than to all tokens in the image. The paper’s qualitative comparison states that the gIoU-versus-GFLOPs curve of HRSeg dominates the naive interpolation baseline.

The method nevertheless retains identifiable constraints. It incurs additional cost for high-resolution crops and for HRP and HRE, especially at $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$03 or $$Y_{\text{seg}} = f_{\text{MLLM}}(I, T), \quad \tilde{Y}_{\text{seg}} = f_{\text{proj}}(Y_{\text{seg}}).$$04. It also depends on SAM proposals: if SAM fails to generate a proposal overlapping the correct object, HRSeg cannot recover it. Failure cases are said to occur in very complex scenes or under highly ambiguous instructions. Future extensions are suggested toward video reasoning segmentation and toward other dense tasks such as detection and instance segmentation (Lin et al., 17 Jul 2025).

The term “HRSeg” also has a broader terminological context. In medical image segmentation, the phrase may refer more generally to an HRNet-style high-resolution segmentation philosophy that keeps a high-resolution branch alive throughout the network and performs repeated multi-resolution fusion. A concrete 3D Transformer instantiation is HRSTNet, the “High-Resolution Swin Transformer Network,” which replaces HRNet’s convolutions with 3D Swin Transformer blocks, maintains a persistent high-resolution branch, and uses Multi-Resolution Feature Fusion for volumetric medical segmentation (Wei et al., 2022). By contrast, HRGS, “Hierarchical Gaussian Splatting,” is not a segmentation method at all but a block-wise framework for memory-efficient high-resolution 3D reconstruction; its appearance in discussions around “HRSeg” reflects lexical similarity rather than task identity (Li et al., 17 Jun 2025). The dominant current use of the exact method name HRSeg, however, is the reasoning-segmentation framework centered on HRP and HRE (Lin et al., 17 Jul 2025).