NL-Refer: Natural Language Reference

• NL-Refer is a family of technologies mapping natural language to stable target representations across symbolic logic, discourse forms, and perceptual grounding.
• It integrates construction-based translation (e.g., SCG), neural referential form selection in discourse (RFS on OntoNotes), and visual grounding (CityFlow-NL) to address diverse reference tasks.
• Practical applications include converting NL into CycL expressions, choosing context-appropriate referential forms, and grounding vehicle descriptions to dynamic tracks in video.

Across the cited literature, NL-Refer can be understood as a family of natural-language reference technologies in which linguistic input is used to construct, select, or ground references to entities across symbolic knowledge bases, discourse models, and perceptual streams. In this sense, the topic spans three closely related problem settings: translation from natural language to logic-like representations suitable for inference and KB access, referential form selection in running discourse, and grounding of natural-language descriptions to objects and trajectories in video. Representative realizations include Semantic Construction Grammar (SCG), which maps surface patterns to CycL expressions (Schneider et al., 2021), multilingual neural Referential Form Selection (RFS) on OntoNotes (Chen et al., 2022), and CityFlow-NL, which grounds descriptions to vehicle tracks and spatio-temporal localizations in city-scale traffic video (Feng et al., 2021).

1. Conceptual scope and task structure

The common core of NL-Refer is the treatment of reference as a mapping from linguistic form to a stable target representation. In SCG, the target is a CycL expression whose denotation can query or extend a knowledge base. In RFS, the target is a referential form class such as proper name, description, pronoun, or, in Chinese, zero pronoun. In CityFlow-NL, the target is either a ranked vehicle track or a frame-level spatio-temporal localization of the described vehicle (Schneider et al., 2021, Chen et al., 2022, Feng et al., 2021).

Paradigm	Input and output	Representative formulation
Symbolic NL-to-logic	NL template $\rightarrow$ CycL expression	SCG constructions
Discourse form selection	$$\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$$	Neural RFS
Visual grounding	NL query $\rightarrow$ ranked tracks or boxes over time	CityFlow-NL

In the discourse setting, RFS is explicitly separated from full REG. The cited formulation treats Referring Expression Generation in context as two steps: Referential Form Selection, which decides whether to use a proper name, description, pronoun, or zero pronoun, and content selection / realization, which decides the internal content of the referring expression. The RFS classifier predicts

$$p\!\left(f \mid x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}; \theta\right)$$

and is trained with cross-entropy over referential-form classes (Chen et al., 2022).

In the visual grounding setting, the two foundational tasks are Vehicle Retrieval by NL and Vehicle Tracking by NL. Retrieval ranks tracks against a query set of three descriptions, while tracking outputs a bounding box or a “not present” decision for each frame. This decomposition makes explicit that NL-Refer is not restricted to static entity linking; it also includes temporal presence detection and continuous localization (Feng et al., 2021).

A plausible implication is that NL-Refer is best viewed not as a single formalism but as a reference-centric design space whose targets differ by domain: denotations in a KB, discourse-salience-conditioned form classes, or physically grounded trajectories.

2. Construction-based symbolic reference

SCG is a construction-based system developed to facilitate translation between natural language and logical representations ranging from forms close to surface structure to KB-oriented forms with higher-order and high-arity relations. Its central objective is to produce “logically composable and inferentially productive logical representations” integrated with Cyc’s knowledge base and inference engine (Schneider et al., 2021).

The basic unit is the construction, which pairs one or more NL templates with exactly one logic template. NL templates consist of strings and typed variables such as $Color#0` or `$PositiveDimensionalThing#0; the logic template is a single CycL expression with corresponding variables. Each construction may also specify anaphoric variables, an output variable, an output type, and positive or negative semantic tests. Matching is type-driven: a span assigned to $Color#0</code> must already have a semantic interpretation whose type is <code>Color</code> or a specialization of <code>Color</code> in Cyc’s <code>isa/genls</code> hierarchy (<a href="/papers/2112.05256" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Schneider et al., 2021</a>).</p> <p>SCG’s examples show that reference construction is not limited to shallow predicate-argument forms. The phrase <strong>“big blue building”</strong> is mapped to $\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$8 and “Barack Obama eats a sandwich” is simplified to $\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$9 after composition and equalSymbols substitution. The phrase “submarine base” is mapped to a KB-native term using Kappa and DeployingMaterialOfTypeFn, illustrating that compact nominal expressions may denote higher-order relational structures rather than merely intersective nominal properties (Schneider et al., 2021).

Semantic constraints are integral rather than post hoc. Positive tests can enforce role compatibility, as in the use of (TypeCapableFn behaviorCapable) to distinguish appropriate senses of “blow out” in “blowing out candles”. Negative tests can exclude semantically implausible readings, as with -(genls $PartiallyTangible#0 SubAtomicParticle) to block an unwanted interpretation of “electron transport”. Global plausibility checks further eliminate type-instance mismatches, disjointness violations, and inter-argument restriction failures (Schneider et al., 2021).

Anaphora is encoded directly in logic templates by allowing typed variables that are absent from the NL side. In the construction for “end of the 2015 season”, $SportsEvent#1 functions as an anaphoric variable and triggers backward search in the parse graph. The current mechanism stops after five possible antecedents or at the beginning of the graph. This makes reference resolution part of the compositional semantics rather than a separate preprocessing stage (Schneider et al., 2021).

The architecture is explicitly non-syntactic in the usual CFG or parse-tree sense. Interpretation proceeds by concept tagging, windowed retrieval of matching constructions, semantic testing, logical composition, simplification, and limited anaphor resolution. This suggests a form of semantically guided parsing in which KB constraints, not phrase-structure derivations alone, are the principal pruning mechanism.

3. Referential form selection in discourse

The OntoNotes-based RFS work treats reference as a classification problem over form classes in realistic multilingual discourse. The dataset is constructed from the English and Chinese portions of OntoNotes, covering six genres: broadcast news, newswire, broadcast conversations, telephone conversations, weblogs, and magazines. The resulting resources are OntoNotes-EN and OntoNotes-ZH, with final sizes of 71,667/8,149/7,619 train/dev/test samples for English and 70,428/9,217/11,607 for Chinese (Chen et al., 2022).

The task definition is explicit. Given a target referent $x^{(r)}$, its pre-context $x^{(\text{pre})}$, and post-context $x^{(\text{post})}$, the model predicts the referential form class $f$. English is evaluated in 4-way, 3-way, and 2-way settings; Chinese is evaluated in 5-way, 4-way, 3-way, and 2-way settings because it includes zero pronoun (ZP) as a full-fledged form class. In the finest-grained setup, English uses $$\{\text{Demonstrative}, \text{Description}, \text{ProperName}, \text{Pronoun}\}$$, whereas Chinese adds ZP (Chen et al., 2022).

A central methodological decision is the use of lexical tags rather than opaque entity IDs. Earlier WebNLG work kept fixed entity tags such as Amatriciana_sauce; the OntoNotes study instead removes underscores and exposes the lexical form, allowing the models to leverage pre-trained embeddings and to generalize to unseen entities. The dataset statistics motivate this choice: only 38.44\% of test referents in OntoNotes-EN and 41.45\% in OntoNotes-ZH appear in training, whereas WebNLG is heavily dominated by seen entities. OntoNotes also differs sharply in discourse profile: first mentions constitute 43\% in both languages, compared with 85\% in WebNLG; proper names constitute 21\% in OntoNotes-EN and 15\% in OntoNotes-ZH, compared with 71\% in WebNLG; and average document lengths are 106.44 and 139.55 tokens for OntoNotes-EN and OntoNotes-ZH, versus 18.62 for WebNLG (Chen et al., 2022).

Three model families are evaluated. The c-RNN concatenates pre-context, target span, and post-context, encodes them with a BiGRU, extracts the hidden states at the beginning and end of the target span, and predicts the form class with a softmax layer. ConATT encodes each segment separately with a BiGRU, applies self-attention to each segment, concatenates the attended segment vectors, and classifies the result. XGBoost serves as a feature-based baseline using linguistically motivated features such as referential status, syntax, and distance (Chen et al., 2022).

The quantitative findings are decisive. On the English 4-way task, c-RNN+BERT reaches macro-F1 74.59, compared with 62.38 for c-RNN and 49.12 for XGBoost. On the Chinese 5-way task, c-RNN+BERT reaches 63.85, compared with 49.62 for c-RNN and 34.59 for XGBoost. The benefit of BERT is larger in Chinese: the relative gain from c-RNN to c-RNN+BERT is +19.57\% in English 4-way and +28.68\% in Chinese 5-way, with the paper reporting average gains of roughly 17.6\% in English versus 24.5\% in Chinese across setups (Chen et al., 2022).

The probing experiments explain part of this performance difference. Probe tasks include discourse status, sentence-level status, syntactic position, distance to antecedent, intervening referent, local prominence, and global prominence. Better RFS performance correlates with stronger encoding of these features, especially in the BERT-based models. The paper further reports that Chinese RFS depends more on discourse context than English, consistent with the heavier contextual load imposed by pronoun versus zero-pronoun selection (Chen et al., 2022).

This work establishes a discourse-oriented form of NL-Refer in which the referent identity is given and the central problem is the mapping from context to how that referent should be realized.

4. Grounding natural-language reference in city-scale video

CityFlow-NL extends the CityFlow benchmark with vehicle-centered natural-language descriptions and formulates NL reference as large-scale visual grounding. The dataset contains 40 calibrated cameras, 666 unique target vehicles, 3,028 single-view tracks, and 5,289 unique descriptions, with an average track length of 75.85 frames per target. For the retrieval task, the single-view split uses 2,498 tracks for training and 530 unique tracks for testing, with 3 NL descriptions per track (Feng et al., 2021).

The annotation protocol has two stages. Workers first assign predefined attributes—vehicle type, color, maneuver/motion, and size—with majority-vote quality control across three annotators. They then write at least three free-form descriptions per track that must uniquely identify the target vehicle. Descriptions may include color and type, maneuver, scene context, and relations to other vehicles, and the instructions explicitly require intrinsic motion to be described. Conflicting descriptions, such as color disagreements under difficult viewing conditions, are retained (Feng et al., 2021).

The retrieval task is formulated in a standard joint-embedding style. A track is $$\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$$0, with cropped target images $$\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$$1 obtained from ground-truth boxes. The visual encoder is ResNet-50 and the text encoder is BERT, projected to $$\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$$2. Frame-query similarity is

$$\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$$3

and track-level similarity for the three-description query set is

$$\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$$4

Tracks are ranked by this score (Feng et al., 2021).

The tracking task is defined as spatio-temporal localization of the described vehicle in a single camera stream. Two baseline strategies are reported. The track-then-retrieve pipeline first detects vehicles with Mask R-CNN, YOLOv3, or SSD512, then forms tracks with DeepSORT, MOANA, or TNT, and finally ranks tracks using the retrieval model. The alternative is the Vehicle Tracking Network (VTN), an end-to-end NL-guided tracker with a Vehicle Presence Network (VPN) and a Vehicle Localization Network (VLN). VPN predicts whether the described vehicle is present in the current frame by cross-correlating the NL representation with FPN features and outputting a presence probability

$$\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$$5

VLN is a modified Faster R-CNN with FPN that adds an NL similarity head, and the full system is trained with

$$\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$$6

At inference time, VTN first thresholds presence and then chooses the highest-scoring region, optionally refined by sub-window attention (Feng et al., 2021).

The benchmark is deliberately difficult. On retrieval, the baseline achieves MRR 0.0269, Recall@5 0.0264, Recall@10 0.0491, and Recall@25 0.1113. On tracking, track-then-retrieve baselines reach AUC Success only between 0.5\% and 2.1\%, whereas VTN reaches 5.93\% AUC Success and 3.79\% normalized precision at threshold 0.5. The qualitative analysis attributes many failures to weak handling of lane-level spatial cues, relations among vehicles, and temporally structured motion descriptions (Feng et al., 2021).

CityFlow-NL therefore instantiates NL-Refer as referring-expression grounding over time: not merely identifying a region in an image, but deciding which trajectory, when it is present, and where it is located frame by frame.

5. Shared representational and algorithmic themes

Despite their different domains, the three strands share several structural commitments. First, they all define a reference target independently of surface wording. In SCG, the target is a logic term or formula with an output variable and type. In RFS, the target referent is fixed and the task is to predict the form class appropriate to that discourse state. In CityFlow-NL, the target is a vehicle track or bounding box sequence scored against one or more descriptions (Schneider et al., 2021, Chen et al., 2022, Feng et al., 2021).

Second, all three rely on typed or structured context rather than raw string matching. SCG uses typed variables, semantic tests, and KB inference over isa, genls, disjointness predicates, and role restrictions. RFS uses pre-context and post-context windows and benefits from models that encode discourse status, syntactic role, and recency. CityFlow-NL couples descriptions to appearance, motion, and scene context over multiple frames and cameras, even though the reported baselines underuse fine-grained relational cues (Schneider et al., 2021, Chen et al., 2022, Feng et al., 2021).

Third, each approach operationalizes reference through a distinct but comparable scoring or derivation mechanism. SCG performs construction retrieval, semantic checking, and logic composition over a parse graph. RFS estimates $$\big(x^{(\text{pre})}, x^{(r)}, x^{(\text{post})}\big) \rightarrow f$$7. CityFlow-NL computes cross-modal similarity in a shared embedding space or performs framewise presence-plus-localization inference (Schneider et al., 2021, Chen et al., 2022, Feng et al., 2021).

A plausible implication is that NL-Refer systems admit a three-level taxonomy. At the symbolic level, reference is resolved into denotations that support inference. At the discourse level, reference is managed by selecting forms consistent with givenness and salience. At the perceptual level, reference is grounded to tracks, regions, and temporal extents. The cited works do not propose a unified architecture across these levels, but taken together they outline a coherent research program.

6. Limitations, controversies, and open directions

The main limitations are domain-specific but structurally related. In SCG, coordination remains difficult, quantification is only partially addressed, anaphora resolution is limited to simple backward search, and large-scale evaluation is hindered by the absence of canonical logical forms. The framework also assumes the availability of the Cyc KB and reasonably accurate concept tagging (Schneider et al., 2021).

In discourse-level RFS, the models are restricted to form selection rather than full REG. The study does not present a joint content-selection model, covers only English and Chinese, and reports that some discourse properties—especially global prominence—are harder for the learned representations to encode. Chinese ZP handling depends on OntoNotes annotations, and the evaluation is comparative rather than truly cross-lingual, since the corpora and label inventories are not parallel (Chen et al., 2022).

In CityFlow-NL, performance remains low in absolute terms, especially for retrieval. The reported models do not explicitly model lane-level spatial structure, inter-object relations, or temporally ordered events, and the paper evaluates only single-view retrieval and tracking even though the underlying benchmark is multi-camera. Annotation subjectivity is also acknowledged: color and other attributes may be described inconsistently across views or lighting conditions (Feng et al., 2021).

Taken together, these limitations indicate that NL-Refer remains constrained by three unresolved problems. One is semantic underspecification, especially for coordination, quantification, and higher-order discourse phenomena. A second is global context modeling, including discourse prominence and long-range antecedent structure. A third is fine-grained grounding, where spatial relations, event order, and multi-entity configurations must be linked to language with substantially greater precision. The cited literature suggests that future progress will likely require tighter integration of symbolic constraints, discourse representations, and cross-modal grounding rather than continued treatment of these as isolated subfields.