NTSFormer: Graph Transformer for Cold-Start Nodes

• The paper introduces NTSFormer, a unified self-teaching framework that produces distinct self-only (student) and self-plus-neighbor (teacher) predictions via a cold-start attention mask.
• NTSFormer employs one-time multimodal graph pre-computation and a position-aware Mixture-of-Experts input projection to effectively handle missing modalities and isolated node structures.
• Empirical results on Movies, Ele-Fashion, and Goodreads-NC demonstrate that NTSFormer outperforms baselines, validating its architectural innovations and efficiency in cold-start scenarios.

Searching arXiv for the cited NTSFormer paper and closely related graph-transformer work. Neighbor-to-Self Graph Transformer (NTSFormer) is a Graph Transformer architecture for multimodal cold-start node classification in which test-time nodes are both structurally isolated and potentially affected by missing modalities. It was introduced as a unified self-teaching framework that avoids degrading cold-start inference to a structure-agnostic MLP, instead producing both a self-only “student” prediction and a self-plus-neighbor “teacher” prediction within the same Transformer through a cold-start attention mask (Hu et al., 7 Jul 2025). In the reported formulation, NTSFormer combines one-time multimodal graph pre-computation, position-aware Mixture-of-Experts (MoE) input projection, and masked self-attention over fixed-length token sequences derived from text, visual, and propagated neighbor features (Hu et al., 7 Jul 2025). The method is positioned at the intersection of cold-start node classification, multimodal learning under missing modalities, and Graph Transformer design, and it also bears a family resemblance to sequence-based graph preprocessing schemes such as Neighbor2Seq, which likewise convert graph neighborhoods into ordered token sequences for mini-batch learning (Liu et al., 2022).

1. Problem setting and research context

NTSFormer addresses multimodal cold-start node classification: given a multimodal attributed graph, the task is to predict labels for newly introduced nodes at test time that are both structurally isolated, meaning they have no edges, and may have missing modalities such as absent text or image features (Hu et al., 7 Jul 2025). This setting is described as common in social networks and e-commerce, for example for new users or products (Hu et al., 7 Jul 2025). The core technical challenge is a conjunction of structural isolation and incomplete content. Isolation prevents exploitation of graph structure by conventional message-passing architectures, while missing modalities reduce the available semantic evidence and complicate multimodal fusion (Hu et al., 7 Jul 2025).

The motivating critique in the NTSFormer paper is directed at two prior lines of work. First, cold-start teacher-student methods such as GLNN, SGKD, and SimMLP train a graph-aware teacher and distill into a structure-agnostic MLP student, which alleviates isolation but reduces inference-time model capacity and requires two-step training and alignment (Hu et al., 7 Jul 2025). Second, missing-modality robustness methods such as MUSE and SiBraR often rely on modality-level mechanisms, while neighbor-based imputation is ineffective when a test node is isolated (Hu et al., 7 Jul 2025). The central claim of NTSFormer is therefore architectural: the student used at cold-start inference should retain full Transformer capacity rather than collapsing to an MLP, and missing-modality robustness should be handled within the same model rather than by an external fallback pipeline (Hu et al., 7 Jul 2025).

This framing also places NTSFormer within a broader trend of transforming graph-structured evidence into sequence form for Transformer-style processing. Neighbor2Seq, for example, formalizes a hop-wise sequence representation $S_i = T(G, i, L) := (z^i_0, z^i_1, \dots, z^i_L)$, where each token aggregates a hop neighborhood, enabling mini-batch training with convolution or attention on precomputed sequences (Liu et al., 2022). NTSFormer adopts a related fixed-length tokenization logic, but specializes it to multimodal cold-start classification with dual self/neighbor branches and modality-aware projections (Hu et al., 7 Jul 2025).

2. Architectural principle: neighbor-to-self self-teaching

The defining mechanism of NTSFormer is a neighbor-to-self self-teaching paradigm implemented inside a single Graph Transformer (Hu et al., 7 Jul 2025). For each node, the model simultaneously produces two predictions by applying a cold-start attention mask over a joint sequence of self tokens and neighbor-context tokens. The “student” prediction uses only self-information, specifically the node’s own text token, image token, and a learnable student classification token $\langle CLS_S \rangle$. The “teacher” prediction incorporates both self tokens and precomputed neighbor-context tokens, together with a learnable teacher classification token $\langle CLS_T \rangle$ (Hu et al., 7 Jul 2025).

The distinction between the two predictions is not realized by separate networks. Instead, both come from the same Transformer under different effective attention permissions, so the teacher supervises the student without an external distillation architecture (Hu et al., 7 Jul 2025). The paper identifies this as a key difference from standard teacher-student pipelines: no separate teacher and student networks are used; both predictions arise from a single masked Transformer; training is end-to-end and single-stage; and the student prediction used at cold-start inference preserves the full Transformer parameterization and tokenization rather than degrading to an MLP (Hu et al., 7 Jul 2025).

The “neighbor-to-self” terminology is apt because the cold-start mask strictly prevents the student branch from reading neighbor tokens while allowing a teacher branch to exploit them. In a different but conceptually adjacent setting, Neighbor2Seq-based Transformer designs also emphasize routing information toward a dedicated self position, using a special self token at position $0$ and attention masks that concentrate information flow through that self representation (Liu et al., 2022). NTSFormer differs in purpose and formal mask definition, but the common structural intuition is that self-centered prediction can be strengthened by sequence-level control of access to neighborhood evidence.

A plausible implication is that NTSFormer reduces train/test mismatch relative to MLP-distillation baselines because the same Transformer parameters and token-processing pipeline are used at training and inference, with the student branch already operating under self-only constraints during training (Hu et al., 7 Jul 2025). The paper states this avoidance of distribution shift explicitly as part of the motivation for the unified design (Hu et al., 7 Jul 2025).

3. Tokenization and multimodal graph pre-computation

NTSFormer begins with one-time multimodal graph pre-computation that converts structural and feature information into fixed-length token sequences (Hu et al., 7 Jul 2025). The model assumes two modalities: text features $X^{(t)} \in \mathbb{R}^{N \times d_t}$ and visual features $X^{(v)} \in \mathbb{R}^{N \times d_v}$, both zero-padded to a shared input dimension $d_{in} = \max(d_t, d_v)$ (Hu et al., 7 Jul 2025). Structural context is derived through multi-hop feature propagation on the symmetrically normalized adjacency:

$$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$

For each hop $k = 1, \dots, K$, neighbor text features $\hat{H}^k X^{(t)}$ and neighbor visual features $\langle CLS_S \rangle$0 are computed (Hu et al., 7 Jul 2025).

Each node is then represented by a concatenated sequence $\langle CLS_S \rangle$1 with total length $\langle CLS_S \rangle$2 (Hu et al., 7 Jul 2025). The self segment contains $\langle CLS_S \rangle$3, where $\langle CLS_S \rangle$4 is a learnable placeholder for a missing modality (Hu et al., 7 Jul 2025). The neighbor segment contains $\langle CLS_S \rangle$5, with $\langle CLS_S \rangle$6 as the teacher classification token (Hu et al., 7 Jul 2025). During training, random modality dropping with probability $\langle CLS_S \rangle$7 simulates missing-modality conditions, although the value is not specified (Hu et al., 7 Jul 2025).

This sequence construction is explicitly modality-aware: text and visual channels are kept distinct across hops, rather than being merged into a modality-agnostic neighborhood encoding (Hu et al., 7 Jul 2025). The paper reports that treating neighbor features as modality-agnostic degrades performance in ablation studies (Hu et al., 7 Jul 2025). This suggests that the tokenization is not a purely engineering convenience, but a substantive inductive bias about the separability of propagated information across modalities.

The design is structurally comparable to Neighbor2Seq’s hop-wise mapping, where sequences are precomputed via repeated sparse multiplications $\langle CLS_S \rangle$8 and used as independent mini-batch samples (Liu et al., 2022). However, NTSFormer extends that paradigm in three directions: separate modality channels, dual classification tokens for self-teaching, and explicit placeholders for missing modalities (Hu et al., 7 Jul 2025).

4. MoE input projection and masked Transformer computation

Because the token sequence mixes semantically diverse elements—self versus neighbor, text versus visual, and special classification or missing-modality tokens—NTSFormer uses a Mixture-of-Experts input projection rather than a single shared projection (Hu et al., 7 Jul 2025). The MoE consists of $\langle CLS_S \rangle$9 routed experts, each a two-layer MLP $\langle CLS_T \rangle$0, plus one shared expert $\langle CLS_T \rangle$1, where the Transformer hidden size is $\langle CLS_T \rangle$2 and the reported default is $\langle CLS_T \rangle$3 (Hu et al., 7 Jul 2025).

Routing is position-aware. For token position $\langle CLS_T \rangle$4, the model concatenates the token feature $\langle CLS_T \rangle$5 with a one-hot position vector $\langle CLS_T \rangle$6, broadcast across all nodes:

$\langle CLS_T \rangle$7

The gating scores are

$\langle CLS_T \rangle$8

with $\langle CLS_T \rangle$9 (Hu et al., 7 Jul 2025). Sparse top-$0$0 routing selects a subset of experts for each node and position, and the routed expert outputs are summed with a shared expert contribution:

$0$1

$0$2

Across all positions, this yields $0$3 (Hu et al., 7 Jul 2025). To avoid expert collapse, the model adds a load-balancing regularizer

$0$4

where $0$5 is the average gate score and $0$6 the fraction of tokens routed to expert $0$7 (Hu et al., 7 Jul 2025).

The Transformer itself is encoder-only, with $0$8 layers, hidden dimension $0$9, and $X^{(t)} \in \mathbb{R}^{N \times d_t}$0 attention heads; the reported defaults are $X^{(t)} \in \mathbb{R}^{N \times d_t}$1 and $X^{(t)} \in \mathbb{R}^{N \times d_t}$2 (Hu et al., 7 Jul 2025). With learnable positional encodings $X^{(t)} \in \mathbb{R}^{N \times d_t}$3 and input $X^{(t)} \in \mathbb{R}^{N \times d_t}$4, each layer follows the standard residual-LayerNorm-FFN pattern:

$X^{(t)} \in \mathbb{R}^{N \times d_t}$5

$X^{(t)} \in \mathbb{R}^{N \times d_t}$6

Masked multi-head attention is defined as

$X^{(t)} \in \mathbb{R}^{N \times d_t}$7

The cold-start mask $X^{(t)} \in \mathbb{R}^{N \times d_t}$8 blocks all attention from the first three self tokens to any neighbor token:

$X^{(t)} \in \mathbb{R}^{N \times d_t}$9

Thus, the student context is strictly self-only, with no leakage from neighbor tokens into $X^{(v)} \in \mathbb{R}^{N \times d_v}$0, while the teacher and neighbor tokens can attend broadly (Hu et al., 7 Jul 2025).

The asymmetry of this mask is central. It does not merely encode sparse structure; it enforces a specific causal partition between self-only and self-plus-neighbor prediction. In Neighbor2Seq-derived masked attention schemes, neighbor-to-self restrictions are also used to focus information flow through a designated self token (Liu et al., 2022). NTSFormer’s mask is more specialized: it is explicitly designed so that one branch simulates cold-start inference while another branch retains graph access for supervision (Hu et al., 7 Jul 2025).

5. Learning objective, missing-modality robustness, and inference

After masked Transformer encoding, NTSFormer extracts a student prediction and a teacher prediction from different token positions (Hu et al., 7 Jul 2025). The student logits are produced from the student classification token:

$X^{(v)} \in \mathbb{R}^{N \times d_v}$1

The teacher logits are produced using the last two tokens:

$X^{(v)} \in \mathbb{R}^{N \times d_v}$2

$X^{(v)} \in \mathbb{R}^{N \times d_v}$3

The training loss combines teacher supervision, self-teaching distillation, and MoE load balancing:

$X^{(v)} \in \mathbb{R}^{N \times d_v}$4

The reported default weights are $X^{(v)} \in \mathbb{R}^{N \times d_v}$5 and $X^{(v)} \in \mathbb{R}^{N \times d_v}$6; no temperature scaling is reported, and there is no student cross-entropy term (Hu et al., 7 Jul 2025).

The missing-modality mechanism operates at token level. Missing features are replaced with the learned $X^{(v)} \in \mathbb{R}^{N \times d_v}$7 token, and random modality dropping during training is used to expose the model to incomplete input conditions (Hu et al., 7 Jul 2025). The self segment is always present, even if some modalities are missing, so student inference remains well-defined (Hu et al., 7 Jul 2025). The paper attributes robustness not only to the placeholder token but also to the MoE routing, which can specialize across self text, self visual, $X^{(v)} \in \mathbb{R}^{N \times d_v}$8-hop text, $X^{(v)} \in \mathbb{R}^{N \times d_v}$9-hop visual, classification, and missing tokens (Hu et al., 7 Jul 2025).

Training and inference are deliberately separated by graph access. During preprocessing, text and visual features may be extracted using encoders such as RoBERTa or T5 for text and CLIP or ViT for images; $d_{in} = \max(d_t, d_v)$0 and $d_{in} = \max(d_t, d_v)$1 are computed offline on CPU without gradients (Hu et al., 7 Jul 2025). During training, graph edges are available for training nodes, and the 60% unlabeled training nodes contribute structure though not labels (Hu et al., 7 Jul 2025). At inference for cold-start nodes, only the self-only sequence $d_{in} = \max(d_t, d_v)$2 is built, with $d_{in} = \max(d_t, d_v)$3 used for absent modalities, and the final prediction is taken from the student head:

$d_{in} = \max(d_t, d_v)$4

The paper evaluates cold-start inference only; while one could use the teacher head for non-cold-start nodes, reported validation and test inference use $d_{in} = \max(d_t, d_v)$5 because those nodes are made isolated (Hu et al., 7 Jul 2025).

A frequent misconception would be to interpret the teacher branch as a conventional online teacher available at deployment. The reported method does not do this. For truly isolated nodes with no usable neighbor context, the teacher branch is not used; student prediction suffices (Hu et al., 7 Jul 2025). The teacher exists as a training-time supervisory view inside the same model, not as a separate deployment component.

6. Empirical evaluation, efficiency, and limitations

The experimental evaluation covers three public datasets under a standardized cold-start protocol (Hu et al., 7 Jul 2025). Movies (MAGB) has 16,672 nodes, 218,390 edges, and 20 classes, with text features from RoBERTa and images from CLIP-ViT. Ele-Fashion (MM-Graph) has 97,766 nodes, 199,602 edges, and 12 classes, with text via T5 and images via ViT-base. Goodreads-NC (MM-Graph) has 685,294 nodes, 7,235,084 edges, and 11 classes, with text via T5-base and images via ViT-base (Hu et al., 7 Jul 2025). The cold-start split is 20% labeled training, 60% unlabeled training, 10% validation as cold-start, and 10% test as cold-start; all edges incident to validation and test nodes are removed to enforce isolation (Hu et al., 7 Jul 2025). Validation and test are each partitioned into Text-Miss, Visual-Miss, and No-Miss subsets, each approximately one-third, while training nodes have complete modalities (Hu et al., 7 Jul 2025).

The reported baselines include MLP; GNNs such as GraphSAGE; multimodal GNNs such as MMGCN, MGAT, and MIG; Graph Transformers such as Polynormer, SGFormer, and NAGphormer; cold-start teacher-student methods such as GLNN, SGKD, and SimMLP; and missing-modality methods such as SiBraR and MUSE (Hu et al., 7 Jul 2025). Accuracy is reported as mean $d_{in} = \max(d_t, d_v)$6 standard deviation over পাঁচ seeds for Text-Miss, Visual-Miss, No-Miss, and All (Hu et al., 7 Jul 2025).

Dataset	NTSFormer result	Best baseline noted
Movies	All: 46.12±0.58	SimMLP All: 43.45±0.88
Ele-Fashion	All: 83.37±0.46	MUSE All: 80.66±0.34
Goodreads-NC	All: 61.58±0.16	SimMLP All: 54.92±0.28

On Movies, NTSFormer reports 45.07±1.12 for Text-Miss, 42.52±1.41 for Visual-Miss, 50.79±1.89 for No-Miss, and 46.12±0.58 overall; the best teacher-student baseline cited is SimMLP at 43.45±0.88 overall (Hu et al., 7 Jul 2025). On Ele-Fashion, NTSFormer reports 80.71±0.89, 83.98±0.80, 85.42±0.54, and 83.37±0.46, while the best missing-modality baseline cited is MUSE at 80.66±0.34 overall (Hu et al., 7 Jul 2025). On Goodreads-NC, NTSFormer reports 50.99±0.13, 63.93±0.36, 69.83±0.47, and 61.58±0.16, compared with SimMLP at 54.92±0.28 overall (Hu et al., 7 Jul 2025).

Ablations reported in the paper support the main architectural claims. Replacing self-teaching with a separate MLP student reduces performance, which the authors interpret as evidence for the advantage of unified Transformer self-teaching (Hu et al., 7 Jul 2025). Removing the MoE input projection and using a single shared linear projection degrades performance, and multiple routed experts with $d_{in} = \max(d_t, d_v)$7 are reported as beneficial (Hu et al., 7 Jul 2025). Treating neighbor features as modality-agnostic harms performance, and the best number of Transformer layers varies across datasets: 2 for Movies, 4 for Ele-Fashion, and 3 for Goodreads-NC (Hu et al., 7 Jul 2025).

The computational profile follows from fixed-length sequence processing. Pre-computing $d_{in} = \max(d_t, d_v)$8 for each modality costs $d_{in} = \max(d_t, d_v)$9, typically $$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$0 with sparse adjacency, and is done once on CPU (Hu et al., 7 Jul 2025). Transformer self-attention operates on length $$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$1, with per-layer/head complexity $$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$2 and overall per-layer cost approximately $$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$3 (Hu et al., 7 Jul 2025). With small $$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$4, such as the reported default $$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$5, this yields $$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$6, so attention cost remains modest (Hu et al., 7 Jul 2025). The default implementation uses $$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$7 routed experts plus one shared expert, top-$$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$8 routing with unspecified $$\tilde{A} = A + I,\quad \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},\quad \hat{H} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}.$$9, dropout 0.2 at input and up to 0.5 in hidden layers, AdamW with learning rate $k = 1, \dots, K$0 and weight decay $k = 1, \dots, K$1, and training schedules of 300 epochs for Movies and Ele-Fashion and 50 for Goodreads-NC, with early stopping on validation accuracy (Hu et al., 7 Jul 2025). On Goodreads-NC, indicative per-epoch training time is reported as 260s for NTSFormer versus 387s for MIG, 778s for GLNN, and 1437s for MUSE, attributed to one-time pre-computation and fixed-length tokenization (Hu et al., 7 Jul 2025).

The limitations stated in the paper are consequential. NTSFormer assumes access to a training graph for pre-computation of $k = 1, \dots, K$2, and fixed modalities with pretrained feature extractors, with zero-padding used to share $k = 1, \dots, K$3 (Hu et al., 7 Jul 2025). Pre-computation may be expensive for highly dynamic graphs, and the propagated features $k = 1, \dots, K$4 capture linear propagations that may miss more complex structural patterns (Hu et al., 7 Jul 2025). The cold-start mask separates student from neighbor tokens at sequence level, but more granular token dependencies such as edge-type-aware modeling are not explored (Hu et al., 7 Jul 2025). Proposed future directions include dynamic graphs, lifelong learning, richer structural tokenization, and adaptive masking or curriculum strategies (Hu et al., 7 Jul 2025).

In the broader literature, NTSFormer can be read as combining three design strands: precomputed graph-to-sequence transformation in the style of Neighbor2Seq (Liu et al., 2022), Transformer-based token interaction under explicit attention constraints, and teacher-student cold-start learning without teacher/student network separation (Hu et al., 7 Jul 2025). Its specific contribution is the integration of these elements into a single Graph Transformer for multimodal cold-start node classification, with self-teaching realized by masking rather than model bifurcation (Hu et al., 7 Jul 2025).