---
title: 'ReMeREC: Multi-Entity REC Framework'
url: https://www.emergentmind.com/topics/remerec
type: topic
---

# ReMeREC: Multi-Entity REC Framework

ReMeREC, short for **Relation-aware and Multi-entity Referring Expression Comprehension**, is a framework for **Referring Expression Comprehension (REC)** that localizes multiple entities in an image from a natural-language description while also predicting directed inter-entity relations. It was introduced together with **ReMeX**, a relation-aware, multi-entity REC dataset, and **EntityText**, an auxiliary text dataset for entity-boundary supervision. The framework targets a limitation of conventional REC: the standard assumption that a description corresponds to a single target region, even though practical expressions often mention multiple entities and depend on action, spatial, and directional relations for disambiguation [2507.16877].

## 1. Task definition and problem scope

Classic REC is formulated as predicting a single region from an image–text pair:
\[
\hat{B} = \mathrm{REC}(I,T).
\]
ReMeREC generalizes this setting to joint multi-entity grounding and relation prediction:
\[
(\hat{\mathbf{B}}, \hat{\mathbf{R}}) = \mathrm{ReMeREC}(I,T).
\]
Here, the output is a set of grounding boxes \(\hat{\mathbf{B}}\) and a set of directed entity relations \(\hat{\mathbf{R}}=\{\hat{r}_{ij}\}\) [2507.16877].

The motivation is that traditional REC methods usually assume one target entity, ignore inter-entity relationships, and require explicit query phrases or fixed target structure. In multi-entity descriptions, however, the model must infer how many entities are mentioned, which text spans correspond to them, and how those entities relate. The paper characterizes this as a shift from isolated grounding to **joint grounding with relational reasoning**. The relevant relations include action relations such as “man holding a laptop,” spatial relations such as “woman next to the car,” and directional relations such as “left of” or “behind.”

A common misconception is that REC difficulty is primarily visual. ReMeREC treats language-side ambiguity as equally central. The paper specifically attributes performance limits in multi-entity REC to **implicit entity boundaries in language**, meaning that the text itself often does not mark entity spans explicitly. This makes entity counting, span extraction, and relation reasoning part of the core task rather than auxiliary preprocessing.

## 2. ReMeX and EntityText as supervision sources

A major part of the contribution is **ReMeX**, a manually annotated benchmark designed for relation-aware multi-entity REC. Images are collected from **Flickr30K** and **COCO**. Annotators first write a caption for each image and then manually annotate bounding boxes, entity labels, and directed relations with the **LabelU** platform; ambiguous or erroneous entries are filtered out [2507.16877].

ReMeX contains **16,530 images**, **16,530 captions**, **23,402 bounding boxes**, and **6,645 relationships**. Each image contains **1 to 4 entity boxes**. Relations are stored as directed subject-to-object pairs through aligned `source` and `target` lists, so the dataset explicitly represents who relates to whom rather than merely which entities are present. This distinguishes ReMeX from classic REC datasets such as **RefCOCO**, **RefCOCO+**, **RefCOCOg**, and **ReferIt**, which primarily support single-entity grounding and generally do not annotate inter-entity relations.

The auxiliary dataset **EntityText** contains **20,000 text annotations** generated using **LLaMA**. Each token is labeled as **1** if it is part of an entity phrase and **0** otherwise. Its purpose is to improve textual understanding, especially token-level entity boundary detection and entity-count estimation. The paper reports that this auxiliary supervision improves entity count prediction accuracy and downstream grounding on ReMeX.

| Resource | Role | Reported statistics |
|---|---|---|
| ReMeX | Relation-aware multi-entity REC benchmark | 16,530 images; 23,402 boxes; 6,645 relationships |
| EntityText | Auxiliary text supervision | 20,000 text annotations; token labels 1/0 |

This pairing of a manually annotated grounding dataset with a token-labeled auxiliary text dataset is significant because ReMeREC treats entity-span discovery as a first-class subproblem. A plausible implication is that the framework depends not only on better multimodal alignment, but also on explicitly structuring the referring expression before visual grounding begins.

## 3. Four-stage architecture

ReMeREC is organized into four stages: **visual and text encoding**, **TMP**, **EIR**, and **query processing / decoding** [2507.16877]. The framework first extracts visual features, textual features, and fused visual-linguistic features. It then identifies entity phrases in the text, aligns them to entity representations, reasons over entity interactions, and finally generates grounding boxes and relations.

The two core modules are **TMP** and **EIR**. **TMP**, the **Text-adaptive Multi-entity Perceptron**, addresses the ambiguity of entity quantity and entity spans in the referring expression. **EIR**, the **Entity Inter-relationship Reasoner**, models interactions among the inferred entities and uses those interactions to improve both relation prediction and grounding consistency.

The architecture is explicitly designed so that entity extraction from text is not a fixed, externally supplied input. Instead, ReMeREC dynamically infers the number of entities, constructs entity-wise representations, and propagates that structure into the multimodal decoder. In that sense, the framework couples textual decomposition and relation-aware grounding more tightly than REC pipelines that process a single phrase independently.

## 4. Text-adaptive Multi-entity Perceptron

TMP is the language-structuring component of ReMeREC. It extracts the **number of entities**, the **span of each entity**, and refined **entity representations** using token-level cues [2507.16877].

Its entity classifier is implemented as an MLP over text features from the context encoder. The penultimate layer produces token-level entity logits, classifying each token as entity or non-entity, while the final layer predicts the entity count. The stage-1 entity loss is
\[
\mathcal{L}_{\mathrm{entity}} = CE(\hat{g}, g) + CE(\hat{N}_e, N_e),
\]
where \(\hat{g}, g\) are the predicted and ground-truth token entity labels and \(\hat{N}_e, N_e\) are the predicted and ground-truth entity counts.

Consecutive tokens above a threshold are grouped into **candidate spans**, with the first and last selected tokens defining the initial start and end positions. Given the predicted entity count, TMP initializes the same number of **learnable entity queries**. These queries are passed through a **Transformer decoder**, where they attend to text features and produce semantic-refined entity representations. The query set is therefore sentence-adaptive rather than fixed a priori.

Because threshold-based span extraction can be imperfect, TMP refines boundaries with a position predictor. The paper defines the centers of estimated and initial spans as
\[
c_{esti} = \frac{s_{esti}+e_{esti}}{2}, \qquad c_{init} = \frac{s_{init}+e_{init}}{2},
\]
and aligns them using Manhattan distance:
\[
\mathbf{C}_{index} = \arg\min \left\| c_{esti} - c_{init} \right\|_{1}.
\]
This step matches a refined query to the nearest candidate span center. TMP also uses an **entity mask** to suppress irrelevant text regions, restricting each entity representation to its own span and reducing contamination from the rest of the sentence.

The role of TMP is therefore broader than phrase extraction. It transforms a single free-form expression into a structured set of entity-wise text representations, each with an inferred span and a dedicated query. The ablations later show that this restructuring materially affects both grounding and relation prediction.

## 5. Entity Inter-relationship Reasoner, decoding, and optimization

EIR receives entity representations from TMP and models their relational structure. It comprises a **relation scoring matrix module**, a **relation count predictor**, and an **entity modulation mechanism** [2507.16877].

The relation scoring matrix fuses TMP entity representations with visual-linguistic features and predicts relation strengths through an MLP. The relation matrix is
\[
\hat{Re} = A^{inter} + A^{sub-obj},
\]
where \(A^{inter}\) denotes interaction affinity scores and \(A^{sub-obj}\) denotes subject-object matching scores. The model also predicts the number of valid relations as an auxiliary constraint. The relation loss is
\[
\mathcal{L}_{\mathrm{relation}} = BCE(\hat{Re}, Re) + CE(\hat{k}, k),
\]
where \(\hat{Re}\) and \(Re\) are the predicted and ground-truth relation matrices and \(\hat{k}, k\) are the predicted and true relation counts. Final relations are obtained by selecting the top-\(\hat{k}\) relations from \(\hat{Re}\).

EIR then feeds relational information back into the entity representations through an entity modulation mechanism. It computes
\[
\mathbf{m} = \text{MLP}(\text{MeanPool}(\hat{Re})),
\]
and updates the entity representation as
\[
\mathbf{Q}^{r} = \mathbf{Q} + \sigma(g)\cdot \mathbf{m},
\]
where \(\mathbf{Q}\) is the original TMP representation, \(\mathbf{Q}^{r}\) is the relation-enriched representation, \(g\) is the gating network output, and \(\sigma(\cdot)\) is sigmoid. This makes each entity representation aware of the global relational structure.

After TMP and EIR, ReMeREC performs query processing and decoding by combining visual-aware text features, entity masks, and entity-relation embeddings. The paper describes attention aggregation for an **entity-context embedding**, concatenation with the **entity-relation embedding**, projection back to the original feature space, addition of a learnable bias embedding, a query decoder with **attention graph convolution**, and cross-attention to decode final multimodal information.

Training is divided into two stages. In **stage 1**, only the context encoder and entity classifier are trained on EntityText with \(\mathcal{L}_{\mathrm{entity}}\). In **stage 2**, the full model is trained on grounding datasets. The bounding-box loss is
\[
\mathcal{L}_{\mathrm{bbox}} = \lambda_{iou}\mathcal{L}_{iou}(\mathbf{B},\hat{\mathbf{B}}) + \lambda_{L1}\|\mathbf{B}-\hat{\mathbf{B}}\|_1,
\]
and the total loss is
\[
\mathcal{L}_{\mathrm{total}} = \lambda_{bbox}\mathcal{L}_{\mathrm{bbox}} + \lambda_{relation}\mathcal{L}_{\mathrm{relation}}.
\]
All weights are set to 1 in the experiments.

## 6. Evaluation, empirical findings, and naming ambiguity

ReMeREC is evaluated on **RefCOCO**, **RefCOCO+**, **RefCOCOg**, **ReferIt**, **ReMeX**, and **Flickr30k Entities**, with **IoU \(\ge 0.5\)** as the grounding criterion. On ReMeX, relation prediction is measured by **image-level relation accuracy**, which requires the full set of predicted relations for an image to exactly match ground truth, and **relation-level accuracy**, which counts an individual predicted relation as correct if it matches any ground-truth relation in that image [2507.16877].

The main reported ReMeX results are **58.32** grounding, **85.74** image-level relation accuracy, and **90.17** relation-level accuracy. In the comparison table reproduced in the summary, these exceed **RefTR** at 36.85 / 62.54 / 75.19, **MDETR** at 39.84 / 65.23 / 75.44, **QRNet** at 42.24 / 74.39 / 81.70, **CLIP-VG** at 50.02 / 80.00 / 83.45, and **HiVG** at 52.03 / 76.78 / 82.66. On classic REC datasets, using RN50-DETR + BERT-base, ReMeREC achieves **89.63 / 91.91 / 86.56** on RefCOCO, **84.31 / 86.29 / 78.89** on RefCOCO+, **86.76 / 87.30** on RefCOCOg, and **76.83** on ReferIt. On Flickr30k Entities, using only the full caption as input, it reaches **62.66**.

The ablations clarify the internal division of labor. Without TMP and EIR, ReMeX performance is **29.45** grounding, **23.54** image-level, and **31.62** relation-level. TMP alone gives **30.38 / 50.87 / 66.59**; EIR alone gives **31.42 / 49.30 / 65.19**; TMP and EIR together give **58.32 / 85.74 / 90.17**. This indicates that language decomposition and relation reasoning are highly complementary. The auxiliary EntityText supervision raises entity count accuracy from **61.46** to **71.74**, grounding from **44.31** to **58.32**, image-level relation accuracy from **76.60** to **85.74**, and relation-level accuracy from **84.75** to **90.17**. Removing relation loss reduces grounding from **58.32** to **42.51**, showing that relational supervision affects localization as well as relation metrics.

These results support the paper’s central claim that standard single-entity REC formulations are inadequate for scenes where entity multiplicity and inter-entity relations matter. A further implication is that relation modeling is not merely an auxiliary head attached to a grounding system; in ReMeREC, it changes the quality of the learned entity representations themselves.

A source of terminology confusion is that similarly named systems exist in unrelated areas. **RecLab** is a distributed offline recommender-system evaluation framework [1810.04957], **RecRec** addresses algorithmic recourse in recommender systems [2308.14916], and **ReRec** is a reasoning-augmented recommendation assistant [2604.07851]. ReMeREC, by contrast, denotes a **vision-language framework for relation-aware multi-entity referring expression comprehension** [2507.16877].

Source: https://www.emergentmind.com/topics/remerec