Where LLM Annotators Fail: Label-Free Learning on Graphs with LLMs

Abstract: Node classification on graphs often requires labeled nodes, yet obtaining labels at graph scale is expensive. When node attributes contain semantic content, such as paper abstracts, web pages, or product descriptions, LLMs can provide low-cost supervision by annotating a small subset of nodes. However, these LLM-generated labels are noisy, and existing label-free graph learning methods usually treat this noise as either global or class-conditional. We find that LLM annotation errors are not only class-dependent but also region-dependent: within the same class, reliability can vary sharply across feature-space clusters. In light of this, we propose Cluster-Aware Noise Estimation (CANE), a label-free learning framework that estimates cluster-conditional LLM reliability without ground truth labels, and uses this estimate to decide which pseudo-labels to trust, and which labels to correct. Across various graph benchmarks and GNN backbones, CANE improves over the strongest label-free baselines, with the largest gains on datasets exhibiting stronger cluster-conditional noise.

• The paper introduces the CANE framework, which models intra-class cluster variations in LLM annotation noise for graph-based learning.
• The paper demonstrates that cluster-aware noise estimation significantly improves classification accuracy, achieving gains up to +6.05pp on benchmarks like WikiCS.
• The paper shows that probe-based noise estimation and iterative label correction offer robust performance even under reduced annotation budgets.

Cluster-Conditional Noise in LLM-Based Label-Free Node Classification

Introduction

The paper "Where LLM Annotators Fail: Label-Free Learning on Graphs with LLMs" (2605.27913) presents a detailed analysis of annotation noise in label-free node classification on text-attributed graphs (TAGs) using LLMs as annotators. The central thesis is that while prior work typically assumes LLM annotation noise is either uniform or class-conditional, in practice, LLM reliability varies sharply within classes—across regions defined by feature-space clusters. This intra-class, region-dependent noise is systematically characterized and exploited by the proposed Cluster-Aware Noise Estimation (CANE) framework, which achieves state-of-the-art label-free node classification accuracy across standard graph benchmarks.

Figure 1: A single class-level accuracy hides large gaps between clusters of the same class, motivating cluster-conditional modeling of LLM reliability.

Cluster-Conditional Noise: Empirical Foundations

Extensive diagnostic analysis demonstrates that class-level aggregate accuracy statistics conceal large intra-class variations. For instance, in DBLP, the "Database" class averages $64.7\%$ LLM annotation accuracy, but when partitioned into feature-space clusters, cluster-level accuracies range from $21.6\%$ to $81.0\%$. This effect is not isolated: across diverse datasets (Cora, CiteSeer, PubMed, WikiCS, DBLP), per-cluster accuracy distributions deviate substantially from what class-conditional noise models predict. ANOVA statistics and raw within-class accuracy gaps confirm strong statistical significance and large effect sizes for local noise structure. Global or class-conditional confusion matrices systematically under-correct in the "hard" clusters—regions of low LLM reliability—where improved error modeling would yield the most benefit.

Figure 2: Per-cell deviation of true per-cluster LLM accuracy from what a global $T$ predicts, illustrating that a global correction under-corrects the hardest clusters.

CANE: Cluster-Aware Noise Estimation Framework

CANE is specifically architected to estimate and utilize cluster-conditional LLM reliability in the absence of ground-truth labels. The pipeline proceeds as follows:

1. Representative Seed Selection: Lacking supervision, a coverage-based, feature-space $K$-means clustering is performed (with $K = 2C$). The densest, most central nodes per cluster are selected as seed nodes, optimizing for representativeness and high LLM annotation accuracy.
2. Probe-Based Noise Estimation: A leading fraction (probe set, default $\rho=0.4$) of the budgeted seeds are used to estimate a three-dimensional transition tensor $T_c \in [0,1]^{K \times C \times C}$. The diagonal $T_c[k,c,c]$ estimates the probability that the LLM labels a class-$c$ node in cluster $21.6\%$0 correctly, using neighbor agreement in both graph and feature spaces as a surrogate for annotation reliability.
3. Pseudo-label Expansion: GNN is trained with early-learning regularization on the annotated seeds. Instead of a static threshold for accepting model predictions as pseudo-labels, the acceptance criterion is a function of the estimated local LLM reliability, i.e., the threshold for a class $21.6\%$1 in cluster $21.6\%$2 is adaptively increased as local reliability decreases (Equation 2 in the paper).
4. Iterative Label Correction: The GNN model is retrained, and label corrections are permitted only when neighboring labels provide sufficient support, with the required support threshold again modulated by the estimated local reliability $21.6\%$3.
5. Final Prediction: The finalized, denoised label set is used for the ultimate GNN training to yield node predictions.

   Figure 3: The CANE pipeline, depicting representative seed selection, cluster-conditional noise estimation, pseudo-label expansion, iterative label correction, and final GNN training.

Experimental Results and Analysis

CANE substantially outperforms leading label-free methods (LLM-GNN [chen2024llmgnn], DMA [sheng2025dma], LoCLE [zhang2025locle]) across five standard benchmarks (Cora, CiteSeer, PubMed, WikiCS, DBLP) and two GNN backbones (GCN, GAT). For high heterogeneity datasets (DBLP, CiteSeer, WikiCS), CANE yields the largest accuracy gains—exceeding previous state-of-the-art by up to $21.6\%$4pp in GAT-based WikiCS—validating the importance of cluster-conditional noise modeling.

Figure 4: Budget sensitivity—CANE maintains accuracy as annotation budgets decrease, while baselines degrade much faster.

Ablation studies confirm that:

• The cluster-conditional noise model ($21.6\%$5) is critical; collapsing it to a class-conditional matrix significantly erodes gains.
• The label-free, agreement-based $21.6\%$6 estimator is on par with oracle (ground-truth-based) versions, with negligible oracle headroom.
• CANE is robust to the choice of clustering, probe fraction, and annotation backbone.
• Accuracy degrades gracefully as the annotation budget is reduced, with CANE showing higher robustness than LoCLE in the low-budget regime.

  Figure 5: Mean accuracy drop from removing each component, demonstrating the importance of the cluster-conditional noise modeling.

  

  Figure 6: Probe-budget ablation showing that the noise estimate stabilizes with moderate probe effort ($21.6\%$7).

  

  Figure 7: Sensitivity to the number of clusters $21.6\%$8; CANE performance is stable across a range of cluster granularities.

Implications and Future Directions

The findings establish cluster-conditional noise as the dominant failure mode of LLM annotators in label-free graph-based learning, challenging reliance on global or class-conditional error models. The proven efficacy of the cluster-aware approach suggests several immediate directions:

• Integration with Graph Structure Modification: While CANE focuses on noise modeling, combining it with graph rewiring (e.g., LoCLE's approach) may yield further improvements, especially for small or highly homophilous graphs.
• Scalability to Large, Weakly-Clustered Graphs: The current formulation relies on meaningful feature-space clusters, and performance degrades for graphs where this structure is absent (e.g., ogbn-arxiv).
• Bias and Fairness Considerations: By quantifying where LLM annotators are least reliable, practitioners gain new leverage for fairness auditing and risk assessment in automated annotation.
• Extension to Heterogeneous or Multimodal Graphs: The methodology is orthogonal to node feature enrichment from LLMs and could be fused with graph–language joint modeling for richer representations.

Conclusion

The paper systematically diagnoses and exploits the cluster-conditional nature of LLM annotation noise in large-scale, label-free node classification. The CANE framework—built on self-supervised cluster-based noise estimation—consistently improves performance over previous approaches, with gains that are tightly coupled to the granularity of local noise structure in the graph. Cluster-aware noise modeling represents an essential refinement for future LLM-assisted graph learning pipelines and offers theoretical and practical insights into the alignment between automated annotation and real network structure.