ReMeREC: Multi-Entity REC Framework

• ReMeREC is a framework for joint multi-entity grounding and relation prediction, addressing conventional REC limitations by inferring entity boundaries and inter-entity dependencies.
• It leverages the Text-adaptive Multi-entity Perceptron (TMP) to extract and refine entity representations along with the Entity Inter-relationship Reasoner (EIR) to model directed interactions.
• The integration of ReMeX and EntityText datasets significantly enhances grounding accuracy and relation prediction compared to traditional single-entity REC approaches.

ReMeREC, short for Relation-aware and Multi-entity Referring Expression Comprehension, is a framework for Referring Expression Comprehension (REC) that localizes multiple entities in an image from a natural-language description while also predicting directed inter-entity relations. It was introduced together with ReMeX, a relation-aware, multi-entity REC dataset, and EntityText, an auxiliary text dataset for entity-boundary supervision. The framework targets a limitation of conventional REC: the standard assumption that a description corresponds to a single target region, even though practical expressions often mention multiple entities and depend on action, spatial, and directional relations for disambiguation (Hu et al., 22 Jul 2025).

1. Task definition and problem scope

Classic REC is formulated as predicting a single region from an image–text pair: $\hat{B} = \mathrm{REC}(I,T).$ ReMeREC generalizes this setting to joint multi-entity grounding and relation prediction: $$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$ Here, the output is a set of grounding boxes $\hat{\mathbf{B}}$ and a set of directed entity relations $\hat{\mathbf{R}}=\{\hat{r}_{ij}\}$ (Hu et al., 22 Jul 2025).

The motivation is that traditional REC methods usually assume one target entity, ignore inter-entity relationships, and require explicit query phrases or fixed target structure. In multi-entity descriptions, however, the model must infer how many entities are mentioned, which text spans correspond to them, and how those entities relate. The paper characterizes this as a shift from isolated grounding to joint grounding with relational reasoning. The relevant relations include action relations such as “man holding a laptop,” spatial relations such as “woman next to the car,” and directional relations such as “left of” or “behind.”

A common misconception is that REC difficulty is primarily visual. ReMeREC treats language-side ambiguity as equally central. The paper specifically attributes performance limits in multi-entity REC to implicit entity boundaries in language, meaning that the text itself often does not mark entity spans explicitly. This makes entity counting, span extraction, and relation reasoning part of the core task rather than auxiliary preprocessing.

2. ReMeX and EntityText as supervision sources

A major part of the contribution is ReMeX, a manually annotated benchmark designed for relation-aware multi-entity REC. Images are collected from Flickr30K and COCO. Annotators first write a caption for each image and then manually annotate bounding boxes, entity labels, and directed relations with the LabelU platform; ambiguous or erroneous entries are filtered out (Hu et al., 22 Jul 2025).

ReMeX contains 16,530 images, 16,530 captions, 23,402 bounding boxes, and 6,645 relationships. Each image contains 1 to 4 entity boxes. Relations are stored as directed subject-to-object pairs through aligned source and target lists, so the dataset explicitly represents who relates to whom rather than merely which entities are present. This distinguishes ReMeX from classic REC datasets such as RefCOCO, RefCOCO+, RefCOCOg, and ReferIt, which primarily support single-entity grounding and generally do not annotate inter-entity relations.

The auxiliary dataset EntityText contains 20,000 text annotations generated using LLaMA. Each token is labeled as 1 if it is part of an entity phrase and 0 otherwise. Its purpose is to improve textual understanding, especially token-level entity boundary detection and entity-count estimation. The paper reports that this auxiliary supervision improves entity count prediction accuracy and downstream grounding on ReMeX.

Resource	Role	Reported statistics
ReMeX	Relation-aware multi-entity REC benchmark	16,530 images; 23,402 boxes; 6,645 relationships
EntityText	Auxiliary text supervision	20,000 text annotations; token labels 1/0

This pairing of a manually annotated grounding dataset with a token-labeled auxiliary text dataset is significant because ReMeREC treats entity-span discovery as a first-class subproblem. A plausible implication is that the framework depends not only on better multimodal alignment, but also on explicitly structuring the referring expression before visual grounding begins.

3. Four-stage architecture

ReMeREC is organized into four stages: visual and text encoding, TMP, EIR, and query processing / decoding (Hu et al., 22 Jul 2025). The framework first extracts visual features, textual features, and fused visual-linguistic features. It then identifies entity phrases in the text, aligns them to entity representations, reasons over entity interactions, and finally generates grounding boxes and relations.

The two core modules are TMP and EIR. TMP, the Text-adaptive Multi-entity Perceptron, addresses the ambiguity of entity quantity and entity spans in the referring expression. EIR, the Entity Inter-relationship Reasoner, models interactions among the inferred entities and uses those interactions to improve both relation prediction and grounding consistency.

The architecture is explicitly designed so that entity extraction from text is not a fixed, externally supplied input. Instead, ReMeREC dynamically infers the number of entities, constructs entity-wise representations, and propagates that structure into the multimodal decoder. In that sense, the framework couples textual decomposition and relation-aware grounding more tightly than REC pipelines that process a single phrase independently.

4. Text-adaptive Multi-entity Perceptron

TMP is the language-structuring component of ReMeREC. It extracts the number of entities, the span of each entity, and refined entity representations using token-level cues (Hu et al., 22 Jul 2025).

Its entity classifier is implemented as an MLP over text features from the context encoder. The penultimate layer produces token-level entity logits, classifying each token as entity or non-entity, while the final layer predicts the entity count. The stage-1 entity loss is

$$\mathcal{L}_{\mathrm{entity}} = CE(\hat{g}, g) + CE(\hat{N}_e, N_e),$$

where $\hat{g}, g$ are the predicted and ground-truth token entity labels and $\hat{N}_e, N_e$ are the predicted and ground-truth entity counts.

Consecutive tokens above a threshold are grouped into candidate spans, with the first and last selected tokens defining the initial start and end positions. Given the predicted entity count, TMP initializes the same number of learnable entity queries. These queries are passed through a Transformer decoder, where they attend to text features and produce semantic-refined entity representations. The query set is therefore sentence-adaptive rather than fixed a priori.

Because threshold-based span extraction can be imperfect, TMP refines boundaries with a position predictor. The paper defines the centers of estimated and initial spans as

$$c_{esti} = \frac{s_{esti}+e_{esti}}{2}, \qquad c_{init} = \frac{s_{init}+e_{init}}{2},$$

and aligns them using Manhattan distance: $$\mathbf{C}_{index} = \arg\min \left\| c_{esti} - c_{init} \right\|_{1}.$$ This step matches a refined query to the nearest candidate span center. TMP also uses an entity mask to suppress irrelevant text regions, restricting each entity representation to its own span and reducing contamination from the rest of the sentence.

The role of TMP is therefore broader than phrase extraction. It transforms a single free-form expression into a structured set of entity-wise text representations, each with an inferred span and a dedicated query. The ablations later show that this restructuring materially affects both grounding and relation prediction.

5. Entity Inter-relationship Reasoner, decoding, and optimization

EIR receives entity representations from TMP and models their relational structure. It comprises a relation scoring matrix module, a relation count predictor, and an entity modulation mechanism (Hu et al., 22 Jul 2025).

The relation scoring matrix fuses TMP entity representations with visual-linguistic features and predicts relation strengths through an MLP. The relation matrix is

$\hat{Re} = A^{inter} + A^{sub-obj},$

where $$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$0 denotes interaction affinity scores and $$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$1 denotes subject-object matching scores. The model also predicts the number of valid relations as an auxiliary constraint. The relation loss is

$$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$2

where $$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$3 and $$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$4 are the predicted and ground-truth relation matrices and $$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$5 are the predicted and true relation counts. Final relations are obtained by selecting the top-$$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$6 relations from $$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$7.

EIR then feeds relational information back into the entity representations through an entity modulation mechanism. It computes

$$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$8

and updates the entity representation as

$$(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).$$9

where $\hat{\mathbf{B}}$0 is the original TMP representation, $\hat{\mathbf{B}}$1 is the relation-enriched representation, $\hat{\mathbf{B}}$2 is the gating network output, and $\hat{\mathbf{B}}$3 is sigmoid. This makes each entity representation aware of the global relational structure.

After TMP and EIR, ReMeREC performs query processing and decoding by combining visual-aware text features, entity masks, and entity-relation embeddings. The paper describes attention aggregation for an entity-context embedding, concatenation with the entity-relation embedding, projection back to the original feature space, addition of a learnable bias embedding, a query decoder with attention graph convolution, and cross-attention to decode final multimodal information.

Training is divided into two stages. In stage 1, only the context encoder and entity classifier are trained on EntityText with $\hat{\mathbf{B}}$4. In stage 2, the full model is trained on grounding datasets. The bounding-box loss is

$\hat{\mathbf{B}}$5

and the total loss is

$\hat{\mathbf{B}}$6

All weights are set to 1 in the experiments.

6. Evaluation, empirical findings, and naming ambiguity

ReMeREC is evaluated on RefCOCO, RefCOCO+, RefCOCOg, ReferIt, ReMeX, and Flickr30k Entities, with IoU $\hat{\mathbf{B}}$7 as the grounding criterion. On ReMeX, relation prediction is measured by image-level relation accuracy, which requires the full set of predicted relations for an image to exactly match ground truth, and relation-level accuracy, which counts an individual predicted relation as correct if it matches any ground-truth relation in that image (Hu et al., 22 Jul 2025).

The main reported ReMeX results are 58.32 grounding, 85.74 image-level relation accuracy, and 90.17 relation-level accuracy. In the comparison table reproduced in the summary, these exceed RefTR at 36.85 / 62.54 / 75.19, MDETR at 39.84 / 65.23 / 75.44, QRNet at 42.24 / 74.39 / 81.70, CLIP-VG at 50.02 / 80.00 / 83.45, and HiVG at 52.03 / 76.78 / 82.66. On classic REC datasets, using RN50-DETR + BERT-base, ReMeREC achieves 89.63 / 91.91 / 86.56 on RefCOCO, 84.31 / 86.29 / 78.89 on RefCOCO+, 86.76 / 87.30 on RefCOCOg, and 76.83 on ReferIt. On Flickr30k Entities, using only the full caption as input, it reaches 62.66.

The ablations clarify the internal division of labor. Without TMP and EIR, ReMeX performance is 29.45 grounding, 23.54 image-level, and 31.62 relation-level. TMP alone gives 30.38 / 50.87 / 66.59; EIR alone gives 31.42 / 49.30 / 65.19; TMP and EIR together give 58.32 / 85.74 / 90.17. This indicates that language decomposition and relation reasoning are highly complementary. The auxiliary EntityText supervision raises entity count accuracy from 61.46 to 71.74, grounding from 44.31 to 58.32, image-level relation accuracy from 76.60 to 85.74, and relation-level accuracy from 84.75 to 90.17. Removing relation loss reduces grounding from 58.32 to 42.51, showing that relational supervision affects localization as well as relation metrics.

These results support the paper’s central claim that standard single-entity REC formulations are inadequate for scenes where entity multiplicity and inter-entity relations matter. A further implication is that relation modeling is not merely an auxiliary head attached to a grounding system; in ReMeREC, it changes the quality of the learned entity representations themselves.

A source of terminology confusion is that similarly named systems exist in unrelated areas. RecLab is a distributed offline recommender-system evaluation framework (Monti et al., 2018), RecRec addresses algorithmic recourse in recommender systems (Verma et al., 2023), and ReRec is a reasoning-augmented recommendation assistant (Huang et al., 9 Apr 2026). ReMeREC, by contrast, denotes a vision-language framework for relation-aware multi-entity referring expression comprehension (Hu et al., 22 Jul 2025).