Rein-G: Domain Generalization for Segmentation

• Rein-G is a parameter-efficient strategy that inserts trainable, instance-aware token modules between frozen backbone layers to achieve robust domain generalization in semantic segmentation.
• It employs low-rank token factorization and additive feature refinements, enabling effective adaptation with less than 1% of backbone parameters being trainable.
• Empirical results demonstrate that Rein-G outperforms full fine-tuning and other PEFT methods across various domain shifts, offering improved mIoU and computational efficiency.

Rein-G is a parameter-efficient fine-tuning strategy for Vision Foundation Models in semantic segmentation under domain generalization. Within the Rein++ framework, it is the domain generalization stage trained only on labeled source domains and designed to generalize to unseen domains without target labels; architecturally, it inserts layer-wise, instance-aware token modules between frozen backbone layers and uses them to compute additive feature refinements while keeping more than 99% of backbone parameters frozen (Wei et al., 3 Aug 2025). The method extends the earlier Rein formulation for domain generalized semantic segmentation, which introduced the same token-based refinement idea but did not use the explicit name “Rein-G” (Wei et al., 2023).

1. Definition, scope, and lineage

Rein-G is defined in Rein++ as the domain generalization solution paired with Rein-A, an unsupervised domain adaptation stage that starts from a Rein-G model and adapts it to unlabeled target domains (Wei et al., 3 Aug 2025). In this formulation, Rein-G learns from labeled source domains only, such as GTAV or Cityscapes, and aims to generalize to unseen domains such as BDD100K, Mapillary, and ACDC without using any target-domain labels or target-domain images during its own training.

Its immediate precursor is Rein, introduced for domain generalized semantic segmentation with Vision Foundation Models. That earlier method already used trainable tokens, low-rank parameterization, layer-wise refinement, and Mask2Former-based decoding, and reported that with “just an extra 1% of trainable parameters within the frozen backbone,” it achieved a Cityscapes mIoU of 78.4 without accessing real urban-scene datasets during training (Wei et al., 2023). A plausible implication is that Rein-G is best understood not as a wholly separate mechanism but as the explicit domain generalization naming and positioning of the Rein method inside a larger two-stage pipeline.

Subsequent application work used Rein without introducing a distinct Rein-G variant. In cross-organ and cross-scanner adenocarcinoma segmentation, Rein was adopted as a fine-tuning method for ConvNeXt and DINOv2, and that paper states that it “do[es] not introduce new technical details beyond those in the original Rein paper” (Cai et al., 2024). By contrast, the name “Rein-G” is also used in an unrelated literature on conversational question answering, where REIGN expands to “REInforced reformulation GeNeration”; that usage refers to question reformulation over knowledge graphs rather than VFM-based segmentation (Kaiser et al., 2023). In the segmentation literature, however, Rein-G denotes the domain generalization component of Rein++.

2. Problem formulation and learning objective

Rein-G addresses two difficulties emphasized in Rein++: the disparity between the scale of segmentation datasets and the much larger corpora used for VFM pre-training, and domain distribution shifts across real deployment scenarios (Wei et al., 3 Aug 2025). The method is explicitly framed for source-only domain generalization rather than domain adaptation. Its goal is to refine a frozen VFM so that its features become better aligned with semantic segmentation while preserving the generalization properties of large-scale pretraining.

Let the pretrained VFM parameters be $\Phi_M$, the segmentation decoder be $\mathcal{H}$ with parameters $\theta_h$, and the Rein-G refinement module have parameters $\theta_R$. The optimization problem is

$$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$

This objective is trained on labeled source data only (Wei et al., 3 Aug 2025). The earlier Rein paper presents the same structure as a frozen VFM backbone plus a trainable decode head and trainable token-based refinement modules, again optimized with standard supervised segmentation loss (Wei et al., 2023).

The conceptual motivation is expressed in terms of a scene gap and a task gap. The scene gap refers to the discrepancy between pretraining imagery and the target deployment imagery; the task gap refers to the discrepancy between pretraining objectives such as contrastive learning or masked image modeling and dense semantic segmentation (Wei et al., 3 Aug 2025). Rein-G addresses both gaps by altering the feature flow through lightweight trainable corrections rather than by updating the backbone weights themselves.

3. Architecture and feature refinement mechanism

Rein-G is VFM-agnostic and has been instantiated on transformer-based backbones including CLIP ViT-L, EVA02-L, DINOv2-L/G, Radio-G, and InternVL-C up to 6B parameters (Wei et al., 3 Aug 2025). Given patch-level features $f_i \in \mathbb{R}^{n \times c}$ at layer $i$, the frozen backbone is modified as

$$\begin{aligned} f_1 &= L_1(Embed(x)),\ f_{i+1} &= L_{i+1}\big(f_i + \Delta f_i\big), \quad i=1,\dots,N-1,\ f_{out} &= f_N + \Delta f_N, \end{aligned}$$

where $\Delta f_i = Rein(f_i)$ is the Rein-G refinement at layer $i$ (Wei et al., 3 Aug 2025). This layer-wise additive correction is the core architectural move: the VFM layers remain frozen, but every layer transition can be modulated by learned, instance-aware corrections.

For each layer, Rein-G introduces a learnable token sequence

$\mathcal{H}$0

with default token length $\mathcal{H}$1 according to ablations (Wei et al., 3 Aug 2025). To reduce parameter count, the token matrix is parameterized in low-rank form:

$\mathcal{H}$2

This low-rank token parameterization is analogous to LoRA in spirit but is applied to token matrices rather than directly to backbone weights (Wei et al., 3 Aug 2025).

The token–feature interaction begins with an attention-like similarity map:

$\mathcal{H}$3

followed by scaled softmax:

$\mathcal{H}$4

Using this similarity map, Rein-G computes a preliminary offset

$\mathcal{H}$5

The method then discards the first token in both $\mathcal{H}$6 and $\mathcal{H}$7 so that each patch is not forced to receive a nonzero modification; this provides an explicit “no-change” option and lets the row sums over effective refinement weights vary in $\mathcal{H}$8 rather than being fixed to $\mathcal{H}$9 (Wei et al., 3 Aug 2025). The final correction uses an MLP with bottleneck and GELU:

$\theta_h$0

The paper also adopts a multi-head mechanism, splitting channels into multiple heads and applying the same operations in parallel without increasing the total parameter budget (Wei et al., 3 Aug 2025).

A further component links the tokens to instance queries for Mask2Former. For each layer,

$\theta_h$1

and the per-layer queries are pooled across layers via max pooling, average pooling, and the final layer output, then concatenated and projected to obtain a single query set $\theta_h$2 for the segmentation decoder (Wei et al., 3 Aug 2025). This query linkage is inherited from the earlier Rein design, where it is presented as “Rein-link” and is used to align token representations with DETR-style object queries (Wei et al., 2023).

4. Optimization regime and parameter efficiency

Rein-G is trained with standard supervised segmentation loss and introduces no additional custom regularizers specifically for domain generalization (Wei et al., 3 Aug 2025). In the Mask2Former formulation, the supervision is the standard mask classification objective with classification loss over query classes together with mask loss terms; the earlier Rein formulation states the optimization in terms of standard supervised segmentation loss as well (Wei et al., 2023).

The trainable parameters are the low-rank token parameters $\theta_h$3 for all layers, the shared MLP weights $\theta_h$4, and the Mask2Former decoder. The pretrained backbone layers and the patch embedding remain frozen (Wei et al., 3 Aug 2025). A key economy is that the MLP weights are layer-shared, while only the per-layer token matrices differ.

Quantitatively, Rein-G tunes a very small fraction of backbone parameters. For DINOv2-L, the trainable backbone parameter counts reported in Table 4 are 304M for full fine-tuning, approximately 0.27M for BitFit, approximately 0.79M for LoRA, 3.17M for AdaptFormer, and 2.99M for Rein-G, with Rein-G achieving the best generalization in that comparison (Wei et al., 3 Aug 2025). For DINOv2-G and Radio-Giant, Rein-G uses approximately 6.36M trainable backbone parameters, and for InternVL-C with 6B parameters it uses approximately 24M, or 0.4% of the backbone (Wei et al., 3 Aug 2025).

The training configuration reported for domain generalization uses MMSegmentation, Mask2Former, AdamW, learning rates of $\theta_h$5 for the backbone and $\theta_h$6 for the Rein-G module and decoder, 40,000 iterations, batch size 4, and $\theta_h$7 crops (Wei et al., 3 Aug 2025). DINOv2-L and EVA02-L DG experiments can be trained in about 12 hours on a single RTX 3090Ti. The efficiency advantage is also reflected in memory and storage: for DINOv2-L on GTAV $\theta_h$8 Cityscapes, full fine-tuning requires 14.7GB GPU memory and 1.22GB of stored trainable parameters, whereas Rein-G uses 9.9GB and 0.09GB; for DINOv2-G, full fine-tuning at batch size 4 is out of memory, whereas Rein-G with batch size 4 uses 24.7GB (Wei et al., 3 Aug 2025).

5. Empirical performance and comparative position

Rein-G is evaluated on multiple source-to-target domain generalization settings: GTAV $\theta_h$9 $\theta_R$0Cityscapes, BDD100K, Mapillary$\theta_R$1, GTAV+Synthia+UrbanSyn $\theta_R$2 $\theta_R$3Cityscapes, BDD100K, Mapillary$\theta_R$4, Cityscapes $\theta_R$5 $\theta_R$6ACDC(val), BDD100K, Mapillary$\theta_R$7, and robustness settings such as Cityscapes $\theta_R$8 ACDC(test) and Cityscapes-C (Wei et al., 3 Aug 2025). The main comparisons are against full fine-tuning, pure freezing, and PEFT baselines such as BitFit, LoRA, AdaptFormer, and VPT.

Backbone / setting	Comparison	Reported average mIoU
DINOv2-L, GTAV $\theta_R$9 $$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$0Citys, BDD, Map$$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$1	Full / Freeze / Rein-G	61.7 / 65.6 / 67.5
DINOv2-L, GTAV+UrbanSyn+Synthia $$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$2 $$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$3Citys, BDD, Map$$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$4	Full / Freeze / Rein-G	67.9 / 70.0 / 71.6
DINOv2-L, Citys $$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$5 $$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$6ACDC(val), BDD, Map$$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$7	Full / Freeze / Rein-G	68.7 / 70.1 / 71.0
Radio-Giant, GTAV $$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$8 $$\mathop{\arg\min}\limits_{\theta_R, \theta_h} \sum_{i=1}^{N_d} \mathcal{L}\big( \mathcal{H}_{\theta_h}\big(\mathcal{F}_{\Phi_M,\theta_R}(x_i)\big), y_i \big).$$9Citys, BDD, Map$f_i \in \mathbb{R}^{n \times c}$0	Full / Freeze / Rein-G	66.4 / 61.0 / 68.8
InternVL-6B, GTAV $f_i \in \mathbb{R}^{n \times c}$1 $f_i \in \mathbb{R}^{n \times c}$2Citys, BDD, Map$f_i \in \mathbb{R}^{n \times c}$3	Freeze / Rein-G	61.1 / 66.8

These figures illustrate two recurrent findings. First, full fine-tuning is not consistently optimal under domain shift; for DINOv2-L, freezing already outperforms full fine-tuning in average mIoU, and Rein-G improves further (Wei et al., 3 Aug 2025). Second, the effect is not limited to moderate-scale backbones: on InternVL-6B, Rein-G yields a 5.7-point absolute improvement over freezing while keeping only 0.4% of backbone parameters trainable (Wei et al., 3 Aug 2025).

Against PEFT baselines on DINOv2-L for GTAV $f_i \in \mathbb{R}^{n \times c}$4 $f_i \in \mathbb{R}^{n \times c}$5Cityscapes, BDD, Mapillary$f_i \in \mathbb{R}^{n \times c}$6, Rein-G reaches 67.5 average mIoU, compared with 65.7 for BitFit, 65.3 for LoRA, 64.8 for AdaptFormer, and 64.4 for VPT (Wei et al., 3 Aug 2025). Against broader domain generalization baselines, Rein-G with Radio-Giant reaches 68.8 average mIoU in the same setting, compared with 55.9 for the best non-VFM DG baseline HRDA and about 68.2 for recent VFM-based DG methods such as SoMA and MFuser; in the multi-source synthetic setting, DINOv2-G with Rein-G reaches 72.5 average mIoU (Wei et al., 3 Aug 2025).

The robustness results are similarly notable. On Cityscapes $f_i \in \mathbb{R}^{n \times c}$7 ACDC(test) and Cityscapes-C, Rein-G achieves 77.6 mIoU on ACDC compared with 67.2 for HGFormer, while attaining comparable average performance on Cityscapes-C and better performance under adverse conditions such as night, snow, fog, and rain (Wei et al., 3 Aug 2025). This suggests that the token-based refinement mechanism improves both out-of-domain generalization and resilience under severe corruptions.

6. Relation to Rein-A, earlier Rein, and interpretive cautions

Within Rein++, Rein-G is the first stage and Rein-A is the second. Rein-A starts from a Rein-G model and performs unsupervised domain adaptation with pseudo-labeling, class-mix, masked-image training, logit- and instance-level adaptation, and a semantic transfer module using the Segment Anything Model (Wei et al., 3 Aug 2025). The importance of the Rein-G initialization is explicit in the ablation cited in Rein++: replacing Rein-G by full fine-tuning in the adaptation stage causes a 5.4 mIoU drop on GTAV $f_i \in \mathbb{R}^{n \times c}$8 Cityscapes, from 76.9 to 71.5 (Wei et al., 3 Aug 2025). Rein-G is therefore not merely a pretraining convenience; it defines the generalizable representation on which Rein-A depends.

Relative to the 2023 Rein paper, Rein-G preserves the same core design principles: a frozen VFM, trainable layer-wise tokens, low-rank token factorization, shared weights across layers, and a query-based segmentation head (Wei et al., 2023). The earlier paper explicitly frames the method as “Leveraging Stronger pre-trained models and Fewer trainable parameters for Superior generalizability,” and reports that full fine-tuning tends to overfit, whereas the token-based refinement improves cross-domain mIoU with about 1% additional trainable backbone parameters (Wei et al., 2023). A plausible reading is that Rein-G formalizes the domain-generalization identity of that method once it is embedded in the broader Rein++ separation between generalization and adaptation.

A common misconception is to treat every use of “Rein” or “Rein-G” as referring to the same method. In digital pathology, Rein is used as a PEFT mechanism for cross-organ and cross-scanner adenocarcinoma segmentation, but that work states that it does not describe a specific Rein-G variant (Cai et al., 2024). In conversational question answering, “Rein-G” may instead refer to REIGN, a reinforcement-learning framework for reformulation generation over knowledge graphs (Kaiser et al., 2023). In the VFM segmentation literature, however, Rein-G specifically denotes the domain generalization module of Rein++.

The limitations stated or implied for Rein-G are also structurally important. The method does not explicitly model style or appearance disentanglement; it does not use target-domain images during the Rein-G stage; its token design remains relatively simple; and its cost scales linearly in patch count $f_i \in \mathbb{R}^{n \times c}$9 and token count $i$0, which can matter in extremely high-resolution segmentation (Wei et al., 3 Aug 2025). Suggested future directions include richer token structures, explicit regularization to enforce instance semantics, combination with style-augmentation DG strategies, and extension to related dense prediction tasks such as panoptic or instance segmentation (Wei et al., 3 Aug 2025).