Relation-Aware LNN-Transformer

• The paper presents a novel architecture merging Continuous-time Forgetting Cell encoders with a bearing-biased self-attention module to capture explicit relational dynamics.
• It employs an intersection graph with static Node2Vec embeddings and sectorized POI descriptors to integrate spatial and temporal features for next-step forecasting.
• Empirical evaluations reveal significant accuracy gains and robust performance under noise, underscoring the benefits of explicit relation-aware design.

Relation-Aware LNN-Transformer denotes a class of Transformer architectures in which relational structure is introduced explicitly into sequence processing, rather than being left entirely to implicit recovery by vanilla self-attention. In the concrete next-step mobility model, the term refers to a hybrid of a Continuous-time Forgetting Cell CfC-LNN and a bearing-biased self-attention module operating on a road-intersection graph, with forecasting restricted to the one-hop candidate set $\mathcal C(v_T)$ of the current intersection (Ren et al., 2 Aug 2025). Related work on multi-person motion, question answering, relational reasoning, and tabular learning shows that the same general design principle can be realized through learned edge tensors, meta-path tokens, cross-modal relative position bias, dual sensory/relational attention, or additive relation matrices (Xu et al., 2023, Park et al., 2022, Altabaa et al., 2024, Konstantinov et al., 8 Dec 2025).

1. Core architectural formulation

In the intersection-centric formulation, the environment is modeled as an undirected road-intersection graph $G=(V,E)$, where each node is a street intersection and each edge is a road segment. A trajectory is represented as $T=(v_1,\dots,v_T)$, and the next-step prediction problem is to infer $v_{T+1}$ from the one-hop candidate set

$\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$

The pipeline combines static node descriptors, online geometric and semantic features, a recurrent temporal encoder, and a Transformer stage with one relation-aware layer (Ren et al., 2 Aug 2025).

The static node representation has two components: a structural embedding $h_v^{\rm struct}\in\mathbb R^{d_s}$ via Node2Vec, and a $168$-dimensional directional POI descriptor $h_v^{\rm POI}\in\mathbb R^{168}$. During online processing, each trajectory step yields a geometric feature $g_t\in\mathbb R^5$ and a POI feature $p_t\in\mathbb R^{168}$; the model uses first-order differences

$G=(V,E)$0

with $G=(V,E)$1. These are encoded by a Continuous-time Forgetting Cell, producing $G=(V,E)$2, and are then merged with the structural prior through a gated mixer before projection into model dimension $G=(V,E)$3 (Ren et al., 2 Aug 2025).

After encoding the observed trajectory, the model performs a final CfC update for each one-hop candidate, conditioning on the geometry from $G=(V,E)$4 to the candidate intersection. The resulting history and candidate tokens are passed through $G=(V,E)$5 standard self-attention layers and $G=(V,E)$6 relation-aware layer, for a total of $G=(V,E)$7. The final embedding $G=(V,E)$8 for each candidate $G=(V,E)$9 is scored by

$T=(v_1,\dots,v_T)$0

The optimization objective combines cross-entropy with an auxiliary direction-regression term,

$T=(v_1,\dots,v_T)$1

The temporal encoder is analytic rather than ODE-solver-based. Its closed-form update is

$T=(v_1,\dots,v_T)$2

with decay factor $T=(v_1,\dots,v_T)$3 and candidate state $T=(v_1,\dots,v_T)$4. The recurrent state is then fused with the structural embedding by

$T=(v_1,\dots,v_T)$5

This recurrent–attentional factorization is the defining architectural feature of the instantiated Relation-Aware LNN-Transformer (Ren et al., 2 Aug 2025).

2. Relation-aware attention as an explicit computational primitive

The relation-aware component in the mobility model is a bearing-biased self-attention layer. For token pair $T=(v_1,\dots,v_T)$6 associated with intersections $T=(v_1,\dots,v_T)$7, the model computes the bearing angle $T=(v_1,\dots,v_T)$8 and forms a head-specific bias

$T=(v_1,\dots,v_T)$9

The weight matrix $v_{T+1}$0 is row-centered for numerical stability so that $v_{T+1}$1. This bias is added to each head’s attention logit, learned type embeddings distinguish history from candidate tokens, and candidate-to-candidate attention is masked so that candidates may attend only to the trajectory (Ren et al., 2 Aug 2025).

An analogous but more elaborate mechanism appears in the Joint-Relation Transformer for multi-person motion prediction. There, attention is conditioned on a learned edge tensor $v_{T+1}$2 whose slices $v_{T+1}$3 encode relative distance and intra-/inter-person physical constraints. A relation-scoring MLP produces $v_{T+1}$4 from $v_{T+1}$5, relation-aware logits augment the usual $v_{T+1}$6 term, and the layer updates both node features and edge features through a joint-relation fusion layer. The node update is followed by a joint-aware relation update using broadcast, concatenation, local MLPs, residual structure, and LayerNorm, so that node and edge embeddings co-evolve rather than being processed in isolation (Xu et al., 2023).

Question Answering Transformer realizes relation awareness differently. It concatenates language-model tokens and knowledge-graph Meta-Path tokens into a single sequence and applies Relation-Aware Self-Attention with a learnable Cross-Modal Relative Position Bias $v_{T+1}$7. The bias depends on matched entity–token pairs identified by cosine similarity in a fixed GloVe space, and it is inserted directly into the self-attention score matrix to guide information exchange between relevant entities of different modalities. No separate GNN or gating network is used; the joint sequence is processed uniformly by each RASA+FFN layer (Park et al., 2022).

The Dual Attention Transformer makes the same separation even more explicit by splitting computation into sensory-attention heads and relational-attention heads. Standard self-attention continues to move sensory information, while the relational branch computes explicit pairwise relation vectors $v_{T+1}$8 and aggregates them together with retrieved symbol vectors. This architecture is presented as a remedy for the claim that the Transformer lacks an explicit computational mechanism for routing and processing relational information (Altabaa et al., 2024).

3. Relation representation and feature carriers

Relation-aware Transformers differ most sharply in how they encode relations before attention. In the road-intersection model, relation awareness is introduced solely through the bearing biases $v_{T+1}$9 in the single relation-aware Transformer layer, while environmental semantics are captured separately through sector-wise directional POI aggregation (Ren et al., 2 Aug 2025). In the Joint-Relation Transformer, by contrast, the relation channel is itself a learned tensor derived from pairwise geometry and physical constraints. In QAT, relations are carried by Meta-Path tokens plus cross-modal bias. In TabRel, they are supplied externally as a symmetric relationship matrix $\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$0 (Xu et al., 2023, Park et al., 2022, Konstantinov et al., 8 Dec 2025).

Model	Relation carrier	Domain
Joint-Relation Transformer	$\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$1 from relative distance, bone adjacency, within-person connectivity	Multi-person motion
QAT	Meta-Path tokens and Cross-Modal Relative Position Bias	Question answering
Relation-Aware LNN-Transformer	Bearing biases and directional POI descriptors	Next-step mobility prediction
TabRel	Symmetric relationship matrix $\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$2 with per-head scalar bias	Tabular learning

The sector-wise directional POI aggregation in the mobility model is unusually concrete. For each intersection, the method fixes a radius $\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$3, partitions the surrounding circle into $\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$4 equal-angle sectors, and aggregates $\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$5 POI categories into a $\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$6-dimensional descriptor. For each category, the summary has $\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$7 dimensions: $\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$8 circular distance/bearing statistics, $\mathcal C(v_T)=\{\,u\mid (v_T,u)\in E\}\,.$9 sector densities, and $h_v^{\rm struct}\in\mathbb R^{d_s}$0 presence flag. The result is z-score normalized across the training set and embedded by a small MLP into the LNN input (Ren et al., 2 Aug 2025).

The JR-Transformer relation tensor is constructed from three raw relation types between any two joints $h_v^{\rm struct}\in\mathbb R^{d_s}$1 and $h_v^{\rm struct}\in\mathbb R^{d_s}$2 at each historical time step $h_v^{\rm struct}\in\mathbb R^{d_s}$3:

$h_v^{\rm struct}\in\mathbb R^{d_s}$4

$h_v^{\rm struct}\in\mathbb R^{d_s}$5

These are stacked into

$h_v^{\rm struct}\in\mathbb R^{d_s}$6

and projected to $h_v^{\rm struct}\in\mathbb R^{d_s}$7 with a $h_v^{\rm struct}\in\mathbb R^{d_s}$8 convolution (Xu et al., 2023).

QAT’s carrier is more symbolic. A $h_v^{\rm struct}\in\mathbb R^{d_s}$9-hop meta-path

$168$0

is treated as a composite relation, and one Meta-Path token embedding $168$1 is constructed per path for $168$2, with $168$3 in experiments. The embedding uses node-type encodings, relation encodings, and a translational difference $168$4 between head and tail entities. This design places relation structure into the token set itself rather than only into the attention bias (Park et al., 2022).

TabRel represents relations most sparsely. In each encoder block, for head $168$5 with raw score matrix $168$6, the model applies

$168$7

where $168$8 is a learnable per-head scalar and $168$9 is a symmetric relationship matrix with $h_v^{\rm POI}\in\mathbb R^{168}$0. Trial-to-trial attention is masked, and the final head predicts only on trial indices (Konstantinov et al., 8 Dec 2025).

4. Supervision, optimization, and training objectives

The instantiated Relation-Aware LNN-Transformer is trained by the sum of a standard cross-entropy term over the correct neighbor and an auxiliary direction-regression loss,

$h_v^{\rm POI}\in\mathbb R^{168}$1

$h_v^{\rm POI}\in\mathbb R^{168}$2

No additional weight-decay or auxiliary regularizers are used beyond standard dropout inside the Transformer, with dropout rate $h_v^{\rm POI}\in\mathbb R^{168}$3. The reported training setup uses Adam with learning rate $h_v^{\rm POI}\in\mathbb R^{168}$4, batch size $h_v^{\rm POI}\in\mathbb R^{168}$5, and $h_v^{\rm POI}\in\mathbb R^{168}$6 epochs; the architecture uses CfC hidden size $h_v^{\rm POI}\in\mathbb R^{168}$7, model dimension $h_v^{\rm POI}\in\mathbb R^{168}$8, and $h_v^{\rm POI}\in\mathbb R^{168}$9 heads, for approximately $g_t\in\mathbb R^5$0M parameters (Ren et al., 2 Aug 2025).

JR-Transformer employs a distinctly multi-objective training signal. Alongside joint-position prediction, it forecasts future relation distances and applies two supervised losses:

$g_t\in\mathbb R^5$1

$g_t\in\mathbb R^5$2

The decoder predicts both $g_t\in\mathbb R^5$3 and $g_t\in\mathbb R^5$4, and deep supervision can be applied at each intermediate pair $g_t\in\mathbb R^5$5 by summing the corresponding losses. This makes the relation channel a directly supervised object rather than merely an internal latent variable (Xu et al., 2023).

QAT is trained with cross-entropy together with a bias regularizer

$g_t\in\mathbb R^5$6

which is subtracted from the cross-entropy so as to encourage positive biases on matched language–KG pairs. The main hyperparameters on CommonsenseQA and OpenBookQA are $g_t\in\mathbb R^5$7, $g_t\in\mathbb R^5$8, $g_t\in\mathbb R^5$9, FFN inner dimension $p_t\in\mathbb R^{168}$0, dropout $p_t\in\mathbb R^{168}$1, and attention-dropout $p_t\in\mathbb R^{168}$2; for MedQA-USMLE, a lighter configuration uses $p_t\in\mathbb R^{168}$3, $p_t\in\mathbb R^{168}$4, $p_t\in\mathbb R^{168}$5, and FFN inner dimension $p_t\in\mathbb R^{168}$6 (Park et al., 2022).

DAT occupies the opposite end of the spectrum. It is trained end-to-end with standard task losses—cross-entropy for classification, teacher-forced cross-entropy for sequence-to-sequence tasks, and next-token cross-entropy for decoder-only language modeling—and explicitly reports that no additional auxiliary losses were needed to induce relational behavior (Altabaa et al., 2024).

5. Empirical behavior, ablations, and robustness

On city-scale road-user trajectories, the Relation-Aware LNN-Transformer is evaluated against six baselines: LSTM, HST-LSTM, STGN, ARNN, MobGT, and GETNext. For full trajectories, the reported results are Acc@1 $p_t\in\mathbb R^{168}$7, Acc@3 $p_t\in\mathbb R^{168}$8, Acc@5 $p_t\in\mathbb R^{168}$9, and MRR $G=(V,E)$00. The best prior, GETNext, attains Acc@1 $G=(V,E)$01, Acc@3 $G=(V,E)$02, Acc@5 $G=(V,E)$03, and MRR $G=(V,E)$04. The relative uplift is reported as $G=(V,E)$05 in Acc@1 and $G=(V,E)$06 in MRR, with perfect top-5 recall at all trajectory lengths (Ren et al., 2 Aug 2025).

The same model is evaluated under two noise regimes. Under Gaussian GPS perturbation with $G=(V,E)$07 up to $G=(V,E)$08 before snapping, Acc@1 drops by only $G=(V,E)$09 percentage points, from $G=(V,E)$10 to $G=(V,E)$11, while Acc@3 and MRR are nearly stable and Acc@5 remains $G=(V,E)$12. Under POI feature noise, each sector count is multiplied by $G=(V,E)$13 with $G=(V,E)$14 and $G=(V,E)$15 up to $G=(V,E)$16; the largest Acc@1 drop occurs at $G=(V,E)$17 with $G=(V,E)$18 percentage points, and the total drop at $G=(V,E)$19 is $G=(V,E)$20 percentage points (Ren et al., 2 Aug 2025).

The ablations are especially informative because they separate the contribution of relation awareness from that of the broader pipeline. Removing POI inputs yields $G=(V,E)$21 percentage points Acc@1; removing Node2Vec structural embeddings yields $G=(V,E)$22 percentage points; removing geometric deltas yields $G=(V,E)$23 percentage points; removing CfC differencing yields $G=(V,E)$24 percentage points; and replacing the relation-aware layer with a vanilla layer yields $G=(V,E)$25 percentage points. This indicates that relation-aware attention contributes measurably, but the overall performance is also strongly conditioned on the directional POI representation and the structural prior (Ren et al., 2 Aug 2025).

JR-Transformer reports consistent gains on multi-person motion forecasting: a $G=(V,E)$26 improvement of $G=(V,E)$27ms VIM on 3DPW-SoMoF/RC and $G=(V,E)$28 improvement of $G=(V,E)$29s MPJPE on CMU-Mpcap/MuPoTS-3D. The paper attributes the gain to explicit geometry, fine-grained attention, two-way coupling between node and edge features, and direct supervision of future inter-joint distances (Xu et al., 2023).

Related relation-aware systems show heterogeneous empirical behavior. QAT reports in-house test accuracy $G=(V,E)$30 on CommonsenseQA versus $G=(V,E)$31 for RoBERTa-only, $G=(V,E)$32 on OpenBookQA with RoBERTa-Large versus $G=(V,E)$33 baseline, $G=(V,E)$34 with AristoRoBERTa QAT, and $G=(V,E)$35 on MedQA-USMLE versus $G=(V,E)$36 for SapBERT-only (Park et al., 2022). DAT reports more than $G=(V,E)$37 absolute better accuracy in small-data synthetic relational games, ViDAT $G=(V,E)$38 on CIFAR-10 versus ViT $G=(V,E)$39, and lower validation perplexity in GPT2-style LLMs, with the same perplexity reached using $G=(V,E)$40–$G=(V,E)$41 fewer tokens (Altabaa et al., 2024). TabRel, however, often underperforms simpler methods in pure regression benchmarks: on the Life Expectancy task, LightGBM attains $G=(V,E)$42 MSE and $G=(V,E)$43 $G=(V,E)$44, whereas TabRel attains $G=(V,E)$45 MSE and $G=(V,E)$46 $G=(V,E)$47 (Konstantinov et al., 8 Dec 2025).

6. Conceptual lineage, scope, and recurring misconceptions

A recurrent misconception is that “relation awareness” requires an explicit graph neural network stage. QAT directly contradicts this: it performs full language–graph integration by concatenating language tokens and KG tokens before every self-attention layer and using Cross-Modal Relative Position Bias, with no separate GNN or gating network (Park et al., 2022). A second misconception is that relation awareness is synonymous with adjacency. In the mobility model, no extra adjacency matrix is needed beyond candidate-neighbor masking; the relation signal is the bearing-based additive bias. In JR-Transformer, the relevant structure is pairwise distance plus physical constraints. In TabRel, the relation input is an externally given symmetric matrix $G=(V,E)$48. This suggests that the operative notion of “relation” in this literature is much broader than graph edge incidence alone (Ren et al., 2 Aug 2025, Xu et al., 2023, Konstantinov et al., 8 Dec 2025).

The literature also shows that relation-aware design is not architecturally uniform. One variant introduces a single specialized attention layer into an otherwise standard stack, as in the mobility model with $G=(V,E)$49 standard layers plus $G=(V,E)$50 relation-aware layer. Another variant treats relations as first-class edge features that are updated jointly with nodes, as in JR-Transformer. A third variant constructs relation-centric tokens, as in QAT’s Meta-Path tokens. A fourth variant separates sensory and relational computation into parallel pathways, as in DAT. This suggests a family resemblance organized around explicit relational computation rather than a single canonical module (Ren et al., 2 Aug 2025, Altabaa et al., 2024).

The acronym “LNN” is not entirely uniform across adjacent discussions. In the instantiated road-intersection model, the architecture is explicitly described as a hybrid of a CfC-LNN and a bearing-biased self-attention module. In the broader relation-aware Transformer literature, Logic Neural Network integration is discussed as an extension: DAT sketches an LNN setting in which symbols, predicate templates, logical masks, and soft logic penalties can be attached to relational heads, and the tabular literature proposes replacing a scalar bias $G=(V,E)$51 with a multi-relation bias derived from learned logical predicates $G=(V,E)$52 (Altabaa et al., 2024, Konstantinov et al., 8 Dec 2025). A plausible implication is that “Relation-Aware LNN-Transformer” names an architectural direction whose unifying feature is explicit relation routing, while the precise meaning of the LNN component depends on whether the model is grounded in continuous-time recurrent dynamics, logical predicate structure, or both.

Within that architectural direction, the principal technical theme is stable across domains: relation information is moved from an implicit by-product of attention weights to an explicit computational object. In mobility forecasting, that object is bearing bias and sectorized semantic context; in motion prediction, it is a learned joint-pair tensor supervised by future distance; in language–graph reasoning, it is Meta-Path structure plus cross-modal bias; in dual-attention models, it is a dedicated relational pathway; and in tabular learning, it is a per-head relation matrix added to attention scores. The shared design claim is therefore not that one relation representation is universally optimal, but that explicit relation channels can be integrated into Transformer computation in a task-specific way (Xu et al., 2023, Park et al., 2022, Altabaa et al., 2024, Ren et al., 2 Aug 2025, Konstantinov et al., 8 Dec 2025).