ReFineG: Low-Resource GMNER Framework

• The paper introduces ReFineG, a hybrid three-stage framework for low-resource GMNER that integrates domain-specific small models with frozen MLLMs.
• It employs synthesized supervision, uncertainty-based refinement, and multimodal retrieval to effectively extract and ground textual entities.
• Empirical results on Twitter-GMNER and CCKS-GMNER show significant F1 improvements, underscoring its value for domain adaptation.

Searching arXiv for the cited ReFineG paper and closely related papers to ground the article. ReFineG is a three-stage collaborative framework for low-resource Grounded Multimodal Named Entity Recognition (GMNER) that combines small supervised models with frozen multimodal LLMs (MLLMs) in order to mitigate scarce multimodal annotations and the Domain Knowledge Conflict observed when generic MLLMs are applied directly to specialized domains (Tang et al., 13 Sep 2025). In the formulation used by the framework, GMNER takes a text–image pair $(T, I)$ and predicts a set of multimodal entities $\{(e_k, c_k, r_k)\}_{k=1}^N$, where $e_k$ is a textual entity span, $c_k$ is its type, and $r_k$ is a 4D bounding box or $\text{None}$ when no visual realization exists. ReFineG is organized as a training stage for domain-aware NER data synthesis, a refinement stage for uncertainty-based collaboration between a small NER model and an MLLM, and a grounding stage for multimodal context selection and analogical reasoning (Tang et al., 13 Sep 2025).

1. Problem setting and motivation

ReFineG is situated in low-resource GMNER, where a system must identify textual mentions and ground them to image regions under limited annotation budgets (Tang et al., 13 Sep 2025). The paper emphasizes four constraints: scarce multimodal annotations, underfitting of supervised models when training data is limited, domain shift between open-domain benchmarks and specialized applications, and the cost of annotation and model adaptation.

A central concept in the framework is Domain Knowledge Conflict. In the paper’s terminology, generic MLLMs are trained on broad web data and, in vertical domains such as military data, often produce redundant generic mentions, incorrect or hallucinated entities, and outputs that do not match domain-specific annotation rules for boundaries, entityhood, or when a region should be set to $\text{None}$ (Tang et al., 13 Sep 2025). The conflict appears especially in textual NER, where the model’s implicit schema diverges from the task schema.

This diagnosis motivates a hybrid division of labor. Small supervised models are used to internalize domain-specific rules, while frozen MLLMs are reserved for cases where broad generalization or visual reasoning is advantageous. This suggests that ReFineG is not a prompt-only MLLM system; rather, it is a collaboration protocol designed to preserve domain alignment while exploiting MLLM flexibility.

2. Three-stage architecture

ReFineG is explicitly defined as a three-stage framework: Training, Refinement, and Grounding (Tang et al., 13 Sep 2025). The small supervised component is a Transformer + CRF NER model, instantiated with BERT-base-uncased or XLM-Roberta-large. The frozen MLLM is Qwen2.5-VL-72B-Instruct. The MLLM is not fine-tuned; domain adaptation occurs through synthesized supervision, uncertainty-based routing, and retrieval-based in-context learning.

Stage	Core mechanism	Output
Training	Domain-aware NER data synthesis	Small supervised NER model
Refinement	Uncertainty-based collaboration	Refined textual entities
Grounding	Multimodal context selection and analogical reasoning	Entity regions or None

In the Training Stage, a text-only LLM builds a global schema guideline from the low-resource labeled set, synthesizes additional NER data through entity-level substitution and sentence-level paraphrasing, and transfers that knowledge into a small supervised NER model (Tang et al., 13 Sep 2025). In the Refinement Stage, the small model predicts entities and computes uncertainty; confident predictions are kept, while uncertain cases are routed to the MLLM for Chain-of-Thought refinement. In the Grounding Stage, the final textual entities are grounded by the MLLM using top-$K$ retrieved multimodal in-context examples selected through entity-, sentence-, and image-level similarity.

The architecture therefore separates textual extraction from visual grounding. The small model is text-only and domain-anchored; the MLLM acts as a selective consultant for textual correction and as the primary grounding engine.

3. Training stage: guideline construction, synthesis, and supervised NER

The training stage begins from a low-resource multimodal labeled set

$$\mathcal{D} = \{(s_i; e_i; v_i)\}_{i=1}^{|\mathcal{D}|},$$

where $s_i$ is a sentence, $\{(e_k, c_k, r_k)\}_{k=1}^N$0 the annotated entities, and $\{(e_k, c_k, r_k)\}_{k=1}^N$1 the paired image with gold bounding boxes (Tang et al., 13 Sep 2025). From this set, ReFineG constructs a global guideline table

$\{(e_k, c_k, r_k)\}_{k=1}^N$2

Here, Typ. is the entity schema, Des. contains descriptions of each type and annotation rules, and Neg. contains LLM-generated negative sample descriptions that capture frequent errors and non-entities.

Two update operations define the calibration loop. Update Neg. compares LLM NER outputs against gold annotations and uses discrepancies such as false positives and boundary errors to refine the negative guidelines. Update Des. asks the LLM to analyze how each entity type is annotated and to refine the type descriptions accordingly (Tang et al., 13 Sep 2025). This procedure is designed to match the actual annotation practice rather than the MLLM’s generic prior.

With $\{(e_k, c_k, r_k)\}_{k=1}^N$3 fixed, the paper uses two synthesis strategies. Entity-level Substitution preserves sentence structure while substituting entity mentions with new ones that conform to the type system and the refined descriptions. Sentence-level Paraphrasing rephrases the sentence while maintaining the same entities and types. The result is a synthetic NER dataset

$\{(e_k, c_k, r_k)\}_{k=1}^N$4

The small supervised model is a standard Transformer + CRF sequence labeler. Given a synthesized sentence $\{(e_k, c_k, r_k)\}_{k=1}^N$5, the probability of a label sequence $\{(e_k, c_k, r_k)\}_{k=1}^N$6 is

$\{(e_k, c_k, r_k)\}_{k=1}^N$7

and training minimizes the negative log-likelihood

$\{(e_k, c_k, r_k)\}_{k=1}^N$8

The encoder is BERT-base-uncased or XLM-Roberta-large, and the CRF enforces structured decoding (Tang et al., 13 Sep 2025).

The implementation details reported for this stage are specific. Training uses AdamW on a single NVIDIA RTX 3090, with learning rate $\{(e_k, c_k, r_k)\}_{k=1}^N$9 for the text encoder, $e_k$0 for the CRF layer, batch size 32, 10 epochs, and maximum sequence length 128 (Tang et al., 13 Sep 2025). These settings reflect the framework’s emphasis on using a relatively small supervised model for domain alignment before invoking the larger frozen MLLM.

4. Refinement and grounding mechanisms

The refinement stage introduces uncertainty-based routing. For each token $e_k$1 in a test sentence $e_k$2, the CRF-based model provides label probabilities $e_k$3 over the label set $e_k$4, and ReFineG defines token-level entropy as

$e_k$5

For an entity mention $e_k$6, entity-level uncertainty is the average token entropy:

$e_k$7

An entity is treated as uncertain when $e_k$8, with $e_k$9 in the reported experiments (Tang et al., 13 Sep 2025).

The decision rule is asymmetric. High-confidence entities are accepted directly from the small model, while high-uncertainty entities are passed, together with the original sentence and paired image, to Qwen2.5-VL-72B-Instruct for Chain-of-Thought refinement (Tang et al., 13 Sep 2025). The prompt instructs the MLLM to reason step by step, analyze why current predictions may be wrong, and correct boundaries, types, or entity presence according to a domain-specific guideline summarized in the prompt. The final textual NER output is the integration of high-confidence small-model predictions and MLLM corrections for high-uncertainty cases.

The grounding stage uses the final textual entities as input to the MLLM and augments the prompt with retrieved multimodal examples. Retrieval is based on three similarity channels computed from separate pretrained encoders: SentenceTransformer all-MiniLM-L6-v2 for text and CLIP ViT-L/14 for images (Tang et al., 13 Sep 2025). The entity-level similarity includes a type-consistency margin:

$c_k$0

where $c_k$1 when entity types match and $c_k$2 (Tang et al., 13 Sep 2025). Sentence-level and image-level similarities are cosine similarities in the corresponding embedding spaces. These scores are combined as

$c_k$3

with $c_k$4, $c_k$5, and $c_k$6, and the top-$c_k$7 examples are selected, with $c_k$8 in the experiments (Tang et al., 13 Sep 2025).

The paper characterizes this as analogical reasoning: the MLLM receives similar annotated examples showing entity mentions, types, and bounding boxes, and is instructed to find the corresponding regions in the target image or return None. This retrieval step is meant to inject domain grounding knowledge without multimodal fine-tuning.

5. Evaluation, baselines, and empirical results

ReFineG is evaluated on Twitter-GMNER and CCKS-GMNER under low-resource settings (Tang et al., 13 Sep 2025). Twitter-GMNER contains 10,000 total samples, 4 entity types, 16,778 entities, and 8,090 regions; the framework follows Li et al. (2024) and uses 10% of the data for training. CCKS-GMNER is a military-domain dataset with 500 annotated instances, 6 entity types, 384 entities, and 307 regions; the reported split uses 100 training instances and 400 test instances to simulate extreme low-resource conditions.

The evaluation protocol is strict. A prediction is counted correct only if the textual mention span, the entity type, and the visual region are all correct. A predicted box is correct if Intersection-over-Union exceeds 0.5, and for non-visual entities the model must output None (Tang et al., 13 Sep 2025). Metrics are Precision, Recall, and F1.

The baselines span three families: fully supervised GMNER systems such as H-index and MQSPN, prompt-only MLLMs such as GPT-4o and Qwen2.5-VL-72B-Instruct, and multimodal data augmentation methods such as MixGen and GMDA (Tang et al., 13 Sep 2025). On Twitter-GMNER with 10% training data, ReFineG achieves F1 = 54.76 with BERT-base and F1 = 57.02 with XLM-R-large, exceeding H-index at 47.46, GMDA(H-index) at 49.23, GPT-4o at 41.29, Qwen2.5-VL-72B at 40.74, and MQSPN at 44.85 (Tang et al., 13 Sep 2025). On CCKS-GMNER with 100 training instances, ReFineG reaches F1 = 72.84 with BERT-base and F1 = 74.76 with XLM-R-large, compared with H-index at 46.33, MQSPN at 49.28, GPT-4o at 64.34, and Qwen2.5-VL-72B at 60.95.

The paper also reports official shared-task results. On the CCKS2025 GMNER online test, the baseline Qwen2.5-VL-72B obtains F1 = 54.59, while ReFineG variants yield F1 = 58.38 without Stage 1, F1 = 61.86 without Stage 2, F1 = 62.37 without Stage 3, and the full ReFineG model reaches Precision 62.61, Recall 66.73, and F1 = 64.61, corresponding to 0.6461 and ranking second on the online leaderboard (Tang et al., 13 Sep 2025).

The ablations isolate the three principal components. Removing synthetic data training (w/o TSD) reduces F1 by 10.21% on Twitter-GMNER and 8.51% on CCKS-GMNER. Removing uncertainty-based refinement (w/o UR) degrades performance consistently. Replacing dynamic multimodal retrieval with fixed 3-shot examples (w/o MES) also lowers F1 (Tang et al., 13 Sep 2025). These results support the paper’s claim that the three stages are complementary rather than interchangeable.

6. Positioning, limitations, and nomenclature

Within multimodal NER, ReFineG is presented as a hybrid alternative to two insufficient extremes: fully supervised multimodal systems that require substantial annotated data, and prompt-only MLLM systems that are vulnerable to Domain Knowledge Conflict (Tang et al., 13 Sep 2025). Its distinctive design choice is the explicit division of responsibilities: the small supervised model serves as the domain-anchored NER core, while the frozen MLLM is used for uncertain textual cases and for visual grounding. A common misconception addressed by the results is that a strong MLLM alone should dominate low-resource GMNER; in the reported experiments, pure MLLM baselines underperform ReFineG on both Twitter-GMNER and CCKS-GMNER (Tang et al., 13 Sep 2025).

The paper also states several limitations. ReFineG depends on the quality of the MLLM, may require re-building guidelines and retrieval encoders for substantially different domains, incurs computational cost because Stages 2 and 3 require multiple calls to Qwen2.5-VL-72B-Instruct and similarity encoders, and is sensitive to prompts, retrieval weights, and the choice of $c_k$9 (Tang et al., 13 Sep 2025). Future work is described in terms of extending the framework to other multimodal information extraction tasks and exploring adaptive strategies for balancing the contributions of small models and MLLMs.

The name ReFineG is not entirely unambiguous across adjacent literature. The explicit use of the title “ReFineG” refers to the GMNER framework described above (Tang et al., 13 Sep 2025). However, the supplied literature also includes EGRefine, an execution-grounded optimization framework for Text-to-SQL schema refinement that treats schemas as objects to be optimized via execution accuracy and SQL views (Wang et al., 1 May 2026). It also includes Refine-n-Judge, a method for automatically curating preference chains for LLM fine-tuning, whose paper states that it is often abbreviated in the paper as “Refine-n-Judge” rather than ReFineG (Cayir et al., 3 Aug 2025). A further related name is ReFine-RFT, a framework for fine-grained visual classification built around Multi-Reward Normalization and reasoning-length control; that paper explicitly notes that it does not define the string “ReFineG” (Zhu et al., 11 Jan 2026). This suggests that, in current arXiv usage, “ReFineG” most precisely denotes the low-resource GMNER framework of (Tang et al., 13 Sep 2025), while neighboring “Refine” names refer to distinct methods in Text-to-SQL, LLM fine-tuning, and FGVC.