Generalization Boundaries of Fine-Tuned Small Language Models for Graph Structural Inference

Abstract: Small LLMs fine-tuned for graph property estimation have demonstrated strong in-distribution performance, yet their generalization capabilities beyond training conditions remain poorly understood. In this work, we systematically investigate the boundaries of structural inference in fine-tuned small LLMs along two generalization axes - graph size and graph family distribution - and assess domain-learning capability on real-world graph benchmarks. Using a controlled experimental setup with three instruction-tuned models in the 3-4B parameter class and two graph serialization formats, we evaluate performance on graphs substantially larger than the training range and across held-out random graph families. Our results show that fine-tuned models maintain strong ordinal consistency across structurally distinct graph families and continue to rank graphs by structural properties on inputs substantially larger than those seen during training, with distinct architecture-specific degradation profiles. These findings delineate where fine-tuned small LLMs generalize reliably, providing empirical grounding for their use in graph-based reasoning tasks.

• The paper demonstrates that fine-tuned SLMs achieve near-perfect in-distribution graph property estimation, particularly with adjacency-list input.
• It reveals that LoRA-based fine-tuning enables robust ordinal inference for out-of-distribution tasks like graph size extrapolation and family transfer.
• The findings imply that while local graph properties are reliably predicted, global combinatorial features remain challenging for current SLM architectures.

Generalization Properties of Fine-Tuned Small LLMs for Graph Structural Inference

Introduction

This essay provides an expert analysis of "Generalization Boundaries of Fine-Tuned Small LLMs for Graph Structural Inference" (2604.18092), focusing on its systematic dissection of the out-of-distribution (OOD) generalization capabilities of fine-tuned small LLMs (SLMs, 3–4B params) for structural property estimation of graphs serialized as text. Unlike most prior work that centers on in-distribution accuracy, this work rigorously explores structural reasoning under distributional shift—scaling graph sizes, transferring between canonical random graph families, and adapting to real-world graph datasets—to delineate the operational trust region for SLMs in graph-theoretic estimation tasks.

Problem Formulation and Methodology

The study evaluates three high-ranking, instruction-tuned SLMs—Llama-3.2-3B, Qwen2.5-3B, Phi-4-mini—parameter-efficiently fine-tuned via LoRA, using two graph input serializations: adjacency-list (Adj) and edge-list (Edge). These LMs are assessed on twelve graph-structural properties, utilizing both synthetic (Erdős–Rényi, Barabási–Albert, Watts–Strogatz) and real-world benchmarks (TUDataset, OGB). The core evaluation axes are (i) graph size extrapolation (testing up to $n=150$ nodes, well beyond the $n=20$–$30$ training regime), (ii) family-wise cross-distribution transfer, and (iii) domain transfer to real data. Macro-averaged Spearman $\rho$, NRMSE$_{\text{range}}$, and $R^2$ are used to capture both ordinal and cardinal prediction fidelity.

Empirical Findings

In-Distribution Calibration

All three SLMs attain near-perfect in-distribution structural ordering and magnitude estimation under adjacency-list encoding (e.g., Qwen2.5-3B: $\rho=0.987$, $R^2=0.972$), outperforming edge-list consistently. This cements text-based SLMs as highly competent graph estimators within their (modest) supervised regime.

Cross-Distribution Transfer

SLMs maintain strong ordinal consistency ($\rho>0.88$—$0.95$) across unseen synthetic graph families under adjacency-list representation, demonstrating abstract, family-agnostic structural reasoning rather than simple memorization of value distributions. Performance (as measured by $n=20$0) is substantially higher for BA and WS transfer targets than ER, indicating the models' increased facility in generalizing to more structurally regular, locally predictable topologies.

Edge-list serialization yields marked reduction in both $n=20$1 and $n=20$2 (e.g., edge-list $n=20$3 frequently falls below 0.3 while adjacency-list remains >0.6), especially when the held-out target features greater topological diversity, revealing the critical dependency of transformer LMs on serialization locality for out-of-domain, global property synthesis.

Size Generalization

Models generalize structural ranking to significantly larger graphs in a graceful, architecture-dependent fashion:

Figure 1: Size generalization performance (macro $n=20$4) as graph size increases, with adjacency-list consistently outperforming edge-list and wider performance gaps at large sizes.

Phi-4-mini achieves monotonic, stable degradation (from $n=20$5 to $n=20$6 at $n=20$7); Qwen2.5-3B exhibits a plateau beyond $n=20$8; Llama-3.2-3B shows non-monotonic, more volatile drop-offs in certain bins. Across the size spectrum, adjacency-list outperforms edge-list (margin widens to >0.1 $n=20$9 at $30$0). These findings directly evidence that (a) transformer-based SLMs encode robust, scalable structural orderings under adjacency-list, and (b) architectural choice critically affects the OOD trajectory of generalization.

Domain Adaptation to Real-World Graphs

With domain-specific fine-tuning (LoRA, $30$1), SLMs yield strong, but less uniform, macro-averaged performance on real graphs:

• On regular molecular graphs (OGB-MolTox21), $30$2 reaches 0.95 (Llama-3.2-3B);
• For structurally diverse datasets (NCI1), performance drops ($30$3 down to 0.68–0.77);
• The adjacency-list margin amplifies for highly irregular biological graphs (ENZYMES), but narrows for regular social networks (IMDB-BINARY), implying that representation format benefits scale non-trivially with topological heterogeneity.

Locality Gradient and Property-Level Analysis

Disaggregated results on ENZYMES exemplify a locality gradient: node-level statistics (degree min/max/mean) are estimated with the highest reliability ($30$4), followed by local-structural (density, clustering), and finally global (diameter, path length; $30$5–$30$6). Cardinally hard, global combinatorial properties (chromatic number) consistently fail (Spearman $30$7), signaling the practical boundary for SLM-based estimation.

Theoretical and Practical Implications

• Generalization Domain: SLMs are reliable for ranking and comparative structural inference tasks—such as screening or heuristic guidance within text-based multi-modal workflows—over considerable OOD shift (family/type/size), provided the serialization is locality-preserving.
• Architectural Sensitivity: The degradation trajectory under OOD extrapolation is highly model-dependent, motivating nuanced architecture selection or ensemble calibration for applications demanding robust OOD performance.
• Representation Dependence: Node-grouped adjacency-list encoding consistently amplifies the attention mechanism's ability to pool local structure, reaffirming that serialization format is not just a nuisance variable but a key design axis for transformer-based graph reasoning.
• Limits of Reasoning: Approximative estimation of properties strongly correlated with local statistics is achievable by SLMs; computation of NP-hard or "globally entangled" invariants remains out of reach with naive text serialization, likely requiring architectural inductive biases or hybrid neuro-symbolic approaches.
• Metric Disentanglement: The observed dissociation between ordinal (Spearman $30$8) and cardinal ($30$9) generalization suggests the existence of separate relational and magnitude estimation circuits in SLMs; downstream tasks must be aligned accordingly to avoid misuse.

Limitations and Future Directions

This investigation is restricted to undirected, unweighted, connected graphs and SLMs in the 3–4B parameter class. Additional inquiry is warranted into (i) directed/weighted/topologically fragmented input, (ii) extrapolation of observed generalization phenomena to larger LMs, and (iii) systematic study of serialization–tokenization interactions, attention head specialization, and representation collapse at extreme OOD scale. Hybrid models integrating symbolic primitive computations or architectural biases (e.g., graph transformers with permutation invariance) may be required to breach the observed inference boundary for global combinatorial properties.

Conclusion

This work systematically characterizes when and how fine-tuned SLMs generalize as approximate graph reasoners under text serialization. Adjacent-list input and explicit parameter-efficient tuning yield state-of-the-art SLM performance for a wide range of structural graph properties, both in- and out-of-distribution. The detailed mapping of generalization boundaries—spanning model, representation, graph size, and domain—provides robust empirical foundations for deploying transformer-based LMs as text-based structural reasoners, while sharply delineating the operational limits of their combinatorial inference capacity.