TMP: Text-adaptive Multi-entity Perceptron

• The paper introduces TMP as a mechanism that transforms a single textual description into distinct entity-specific outputs including count, spans, and embeddings.
• TMP employs token-level classification and learnable queries within a Transformer decoder to precisely identify and represent multiple entities.
• Empirical results show that integrating TMP with relational reasoning yields significant grounding performance gains in multi-entity referring expression comprehension.

Searching arXiv for the specified paper to ground the article in the original source. I’m unable to access the arXiv search tool in this session, and the writing instructions require arXiv-grounded sourcing. Please enable the tool or provide any additional source excerpt if you want a strictly tool-backed version.

Text-adaptive Multi-entity Perceptron (TMP) is a component of the ReMeREC framework for multi-entity referring expression comprehension, introduced in “ReMeREC: Relation-aware and Multi-entity Referring Expression Comprehension” (Hu et al., 22 Jul 2025). Its function is to transform a single natural-language description containing an unknown number of entity mentions into a structured set of entity-specific outputs: an inferred entity count, token-span boundaries for each entity phrase, and distinct per-entity embeddings. Within ReMeREC, TMP serves as the text-side front end that bridges sentence-level language understanding and downstream inter-entity relational reasoning. The design addresses a central difficulty of multi-entity REC: unlike classic REC, where each query phrase and entity span are given, multi-entity descriptions present multiple entities implicitly within one sentence, so both segmentation and representation must be inferred jointly (Hu et al., 22 Jul 2025).

1. Problem Setting and Functional Role

In classic REC, each query phrase and its entity span are given. In multi-entity REC, by contrast, the textual description is a single sentence containing an unknown number of entity phrases, exemplified by expressions such as “the man holding a laptop and the dog to his left” (Hu et al., 22 Jul 2025). TMP is introduced to resolve this ambiguity at the language interface.

Its job is explicitly two-fold. First, it infers how many entities the sentence refers to. Second, it locates the token-span for each entity phrase and produces a distinct embedding for each (Hu et al., 22 Jul 2025). These outputs are not auxiliary annotations; they define the interface between language parsing and the subsequent Entity Inter-relationship Reasoner (EIR). In that sense, TMP supplies both entity identity at the token level and entity semantics at the embedding level.

The broader significance of TMP lies in its treatment of implicit entity boundaries in language. The ReMeREC formulation identifies semantic ambiguity arising from the fact that a sentence does not explicitly delimit each entity mention in a form directly usable by downstream grounding modules (Hu et al., 22 Jul 2025). TMP is therefore designed as a structured parsing-and-representation mechanism rather than a generic sentence encoder. A plausible implication is that the module is intended to make entity decomposition an explicit learned subproblem instead of leaving it to latent cross-modal attention alone.

2. Inputs, Outputs, and Internal Architecture

TMP takes as input text features

$T = [t_1,\dots,t_L] \in \mathbb{R}^{L \times d}$

from the context encoder, described as a BERT-based Transformer (Hu et al., 22 Jul 2025). Visual-lingual fused features $V_\ell$ from the cross-modal encoder may optionally be used later to re-score spans, but they are not required for the core TMP span extraction (Hu et al., 22 Jul 2025). This text-first design is important because the module’s primary function is to infer entity structure directly from fine-grained textual cues.

TMP consists of three sub-modules: an Entity Classifier, Learnable Entity Queries, and a Position Predictor (Hu et al., 22 Jul 2025). The Entity Classifier is a two-stage feed-forward network applied to pooled text features. Its penultimate layer produces token-level “entity logits” $\hat{g} \in \mathbb{R}^{L \times 2}$, classifying each token representation $t_i$ as entity or non-entity. Its final layer aggregates over $T$, for example via mean-pool, and outputs the predicted number of entities, $\hat{N}_e$ (Hu et al., 22 Jul 2025). Token logits are thresholded, with the description giving 0.5 as an example, to form candidate spans

$$\{[s_{\text{init}}^j,e_{\text{init}}^j]\}_{j=1\dots M}$$

of consecutive entity tokens.

On the basis of the predicted entity count, TMP initializes a set of learnable entity queries $Q = \{q_i^0\}_{i=1\dots \hat{N}_e}$, with each

$q_i^0 \in \mathbb{R}^d$

a learned vector (Hu et al., 22 Jul 2025). These queries, together with the text features $T$, are passed through a lightweight Transformer decoder with two layers. The decoder cross-attends the initial queries to the sentence representation and yields refined query embeddings $V_\ell$0, each intended to encode the semantics of one entity (Hu et al., 22 Jul 2025).

The Position Predictor maps each refined query through a small MLP to estimate normalized start and end positions, $V_\ell$1, which are then scaled to token indices (Hu et al., 22 Jul 2025). For each matched span, TMP builds an entity mask $V_\ell$2 that zeros out tokens outside the selected token interval. The system may then re-pool or re-attend over the masked text to obtain a final per-entity textual embedding (Hu et al., 22 Jul 2025).

The resulting outputs are the inferred number of entities $V_\ell$3, refined entity embeddings $V_\ell$4, text-span boundaries $V_\ell$5, and entity masks $V_\ell$6 (Hu et al., 22 Jul 2025).

3. Core Mathematical Formulation

The main computations in TMP are defined explicitly in the ReMeREC description (Hu et al., 22 Jul 2025). For entity count prediction and token-level classification, the module uses

$V_\ell$7

$V_\ell$8

Here, $V_\ell$9 provides tokenwise entity/non-entity logits, while $\hat{g} \in \mathbb{R}^{L \times 2}$0 estimates the number of entity mentions contained in the sentence.

For each query embedding $\hat{g} \in \mathbb{R}^{L \times 2}$1, the Position Predictor estimates normalized boundaries:

$\hat{g} \in \mathbb{R}^{L \times 2}$2

followed by rescaling to token indices,

$\hat{g} \in \mathbb{R}^{L \times 2}$3

These predicted boundaries are not used in isolation. Instead, TMP computes geometric centers for both the predicted span and each threshold-derived candidate span:

$\hat{g} \in \mathbb{R}^{L \times 2}$4

Matching is then performed by assigning each query to the closest candidate span in $\hat{g} \in \mathbb{R}^{L \times 2}$5 center distance:

$\hat{g} \in \mathbb{R}^{L \times 2}$6

This center-matching mechanism couples continuous boundary prediction with discrete candidate spans extracted from token logits (Hu et al., 22 Jul 2025).

After masking, a final entity embedding may be produced as

$\hat{g} \in \mathbb{R}^{L \times 2}$7

This formulation indicates that the final entity representation can be refined by re-attending to the text restricted to the matched entity span (Hu et al., 22 Jul 2025). A plausible implication is that TMP uses coarse-to-fine localization: first identifying likely entity token regions, then consolidating a query-conditioned representation over that localized text.

4. Algorithmic Flow

The step-by-step procedure of TMP is described explicitly in the ReMeREC documentation (Hu et al., 22 Jul 2025). Given token embeddings $\hat{g} \in \mathbb{R}^{L \times 2}$8, the module first computes token logits and the predicted entity count:

$\hat{g} \in \mathbb{R}^{L \times 2}$9

and $t_i$0. The token logits are thresholded to obtain candidate spans $t_i$1.

Next, TMP initializes $t_i$2 learnable queries $t_i$3 and feeds them, together with $t_i$4, into a Transformer decoder to obtain refined query embeddings $t_i$5 (Hu et al., 22 Jul 2025). Each query then passes through the position predictor, which estimates normalized boundaries $t_i$6, rescales them to the token axis, computes the predicted center, and matches that center to the closest initial candidate span center. From the matched span, TMP builds an entity mask $t_i$7 and may optionally re-attend to the masked sentence representation to obtain $t_i$8 (Hu et al., 22 Jul 2025).

The algorithm therefore combines two sources of structure: tokenwise entity segmentation from the classifier and query-conditioned refinement from the decoder. This suggests that TMP avoids relying exclusively on either sequence labeling or slot-based decoding. Instead, it uses thresholded token predictions to generate candidate entity regions, then resolves these regions through a fixed number of learned entity queries determined by the predicted count.

Figures referenced in the ReMeREC paper show the overall pipeline with TMP boxed, the internal three-block flow of TMP, and the span-matching process via center distances (Hu et al., 22 Jul 2025). Even without reproducing those figures, the textual specification makes clear that TMP is not merely an extractor of spans but also a constructor of entity-indexed latent variables intended for subsequent relational modules.

5. Training Objectives and Optimization Regime

ReMeREC is trained in two stages, and TMP is pre-trained in Stage 1 on the auxiliary dataset EntityText (Hu et al., 22 Jul 2025). In that stage, the supervision directly targets the two central TMP outputs: token-level entity membership and entity count. Specifically, TMP uses cross-entropy on token logits, $t_i$9, where

$T$0

contains ground-truth entity/non-entity labels, and cross-entropy on count, $T$1. These combine into the entity loss

$T$2

During this stage, all other modules are frozen; only the context encoder and entity classifier are trained (Hu et al., 22 Jul 2025).

In Stage 2, TMP continues to be optimized jointly under the full grounding and relation losses, but no extra regularization is introduced specifically for TMP (Hu et al., 22 Jul 2025). This training design places TMP in a hybrid role. It is first established as a text-only entity parser under explicit supervision, then integrated into a larger multimodal system where its parameters continue to adapt to downstream grounding and relation objectives.

The use of EntityText is also notable. ReMeREC describes it as a small-scale auxiliary dataset generated using LLMs to improve language comprehension for fine-grained prompts (Hu et al., 22 Jul 2025). Since TMP is the primary text decomposition mechanism in the system, the Stage 1 pre-training protocol directly addresses its need for high-quality supervision on entity segmentation and cardinality.

6. Interaction with Entity Inter-relationship Reasoner

TMP is tightly coupled to the Entity Inter-relationship Reasoner (EIR), which is the module responsible for relational reasoning and global scene understanding in ReMeREC (Hu et al., 22 Jul 2025). TMP delivers the refined entity embeddings

$T$3

and spans $T$4. EIR then fuses each entity representation with visual-lingual features at corresponding image regions:

$T$5

The fused representations $T$6 are used to build the inter-entity relation score matrix

$T$7

and to compute modulation scores $T$8 for each entity (Hu et al., 22 Jul 2025).

This division of labor is conceptually precise. TMP identifies “who” the entities are in the sentence and “what” textual semantics each entity embedding should carry, while EIR reasons about “how” these entities relate (Hu et al., 22 Jul 2025). The system design therefore separates entity decomposition from inter-entity reasoning while preserving a formal interface between them.

The reported behavior of the overall framework indicates that the two modules are complementary rather than redundant. TMP alone improves multi-entity localization only modestly, but its interaction with EIR yields substantially larger gains (Hu et al., 22 Jul 2025). This suggests that entity parsing and relational inference are interdependent in multi-entity REC: accurate spans provide cleaner entity semantics for relation modeling, and relation modeling, in turn, can exploit that decomposition to resolve visually ambiguous scenes.

7. Empirical Effects and Interpretation

The ablation results on the ReMeX dataset quantify the contribution attributed to TMP within ReMeREC (Hu et al., 22 Jul 2025). The paper reports the following grounding results from Table 3:

Configuration	Grounding
Base w/o TMP, w/o EIR	29.45%
+TMP only	30.38%
+EIR only	31.42%
+TMP and +EIR	58.32%

These numbers show that TMP alone increases grounding from 29.45% to 30.38%, a gain of +0.93, while EIR alone increases it to 31.42%, a gain of +1.97 (Hu et al., 22 Jul 2025). When both are present, grounding reaches 58.32%. The source description characterizes this as “huge synergy” (Hu et al., 22 Jul 2025). In stricter encyclopedic terms, the data indicate a strong non-additive interaction between the two modules.

EntityText pre-training also improves TMP’s count accuracy from 61.46% to 71.74%, and this change is reported to translate into overall grounding gains from 44.31 to 58.32% (Hu et al., 22 Jul 2025). This connects the local text-side task of count prediction to end-to-end grounding performance. A plausible implication is that entity cardinality estimation is not a peripheral subtask but a structural determinant of downstream performance, because it defines how many entity queries are instantiated and therefore how subsequent relation and grounding computations are organized.

A common misconception would be to regard TMP simply as a span detector. The ReMeREC specification does not support that reduction. TMP predicts token-level entity membership, infers the number of entities, initializes and refines learned entity queries, predicts span positions, constructs masks, and optionally re-pools masked text into final per-entity embeddings (Hu et al., 22 Jul 2025). Its role is therefore broader: it is a text-adaptive mechanism for decomposing a global sentence into a set of entity-specific latent representations suitable for multimodal reasoning.

In summary, TMP is the text-only front end of ReMeREC that turns one global sentence into $T$9 separate entity queries, finds their token spans, and embeds each entity independently, thereby enabling both precise multi-entity grounding and subsequent inter-entity relationship reasoning (Hu et al., 22 Jul 2025).