HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy

Abstract: Cross-document relation extraction (RE) aims to identify relations between the head and tail entities located in different documents. Existing approaches typically adopt the paradigm of ``\textit{Small LLM (SLM) + Classifier}''. However, the limited language understanding ability of SLMs hinders further improvement of their performance. In this paper, we conduct a preliminary study to explore the performance of LLMs in cross-document RE. Despite their extensive parameters, our findings indicate that LLMs do not consistently surpass existing SLMs. Further analysis suggests that the underperformance is largely attributed to the challenges posed by the numerous predefined relations. To overcome this issue, we propose an LLM-based \underline{H}ierarchical \underline{C}lassification model for cross-document \underline{RE} (HCRE), which consists of two core components: 1) an LLM for relation prediction and 2) a \textit{hierarchical relation tree} derived from the predefined relation set. This tree enables the LLM to perform hierarchical classification, where the target relation is inferred level by level. Since the number of child nodes is much smaller than the size of the entire predefined relation set, the hierarchical relation tree significantly reduces the number of relation options that LLM needs to consider during inference. However, hierarchical classification introduces the risk of error propagation across levels. To mitigate this, we propose a \textit{prediction-then-verification} inference strategy that improves prediction reliability through multi-view verification at each level. Extensive experiments show that HCRE outperforms existing baselines, validating its effectiveness.

• The paper introduces a hierarchical classification paradigm that incrementally narrows candidate relations, reducing prompt length and cognitive load.
• The paper employs a prediction-then-verification strategy to mitigate error propagation, boosting accuracy by over 7 micro F1 points.
• The paper demonstrates state-of-the-art performance on CodRED, outperforming traditional methods with robust results across multiple LLM backbones.

HCRE: Hierarchical Classification with LLMs for Cross-Document Relation Extraction

Problem Motivation and Limitations of Current Approaches

Cross-document relation extraction (RE) addresses the challenge of discovering relational facts between entities that do not co-occur within the same document, modeling complex, long-range dependencies. Previous approaches decompose the problem with SLM+classifier pipelines, which, due to the sheer number of candidate relations and class imbalances, achieve limited performance, especially when entity pairs require disambiguation among hundreds of fine-grained relations.

Recent attempts to directly incorporate LLMs into this setting, leveraging their strong contextual capabilities, fail to consistently outperform SLM-based baselines. This underperformance is traced to two factors: (1) the large candidate relation set (e.g., 277 relations in CodRED), which both increases prompt length and demands nuanced discrimination between semantically similar relations, and (2) the presence of a large proportion of negative (NA) cases. Empirical results consistently reveal poor micro F1 and binary F1 when using conventional LLM prompting.

Hierarchical Paradigm for LLM-based Cross-Document RE

To overcome the limitations of direct classification, the hierarchical classification paradigm is introduced, as illustrated in Figure 1. Unlike direct selection from a flat relation set, HCRE organizes relations as a hierarchical tree, with internal nodes representing coarse-grained concepts and leaves as fine-grained atomic relations.

Figure 1: Comparison of direct classification (baseline) and hierarchical classification (HCRE) in cross-document RE; hierarchical classification leverages a relation tree for progressive narrowing of choices.

Rather than exposing the LLM to the full set of candidate relations at once, inference proceeds top-down—at each hierarchical level, the LLM is prompted with only a handful of child concepts, incrementally focusing its attention and reasoned discrimination. This drastically reduces the cognitive burden per inference step and decreases prompt length, as substantiated by the dramatic improvement in LLM classification performance as the candidate set is pruned (Figure 2).

Figure 2: LLM accuracy improves as the set of candidate relations in the prompt is reduced, demonstrating the impact of prompt length and option set size on performance.

The relation tree itself is constructed automatically by a second LLM (e.g., GPT-4o), based on high-level relation semantics, using level-wise semantic partitioning (Figure 3). This process produces a tree with low inter-node ambiguity and supports robust partitioning even under variable random seeds and partitioning criteria.

Figure 3: The level-wise pipeline for hierarchical relation tree construction using LLM-based semantic clustering with multi-level partitioning.

Prediction-then-Verification (PtV) Inference Strategy

Hierarchical classification introduces the risk of error propagation: errors in early-stage node selection can irreversibly eliminate correct leaf relations. HCRE addresses this with a prediction-then-verification (PtV) inference strategy (Figure 4). At each level:

1. The LLM produces a ranked selection (best node and runner-up) from among the child nodes.
2. Verification is performed by constructing alternative prompts in which these selected nodes are replaced by their children. The LLM is then queried on each, producing multiple “perspectives” on the correct choice.
3. The original choice is retained only if most verifications are consistent (i.e., supporting the chosen path or its descendants). Otherwise, the suspect branch is pruned and the process is repeated.

   Figure 4: Overview of HCRE. (a) Hierarchical classification guided by the relation tree. (b) Multi-view prediction-then-verification mitigates error propagation across levels.

The predictive refinement achieved by PtV sharply reduces the incidence of false pruning—quantitative analysis (Figure 5) demonstrates substantial improvements in node-level accuracy and a lower ratio of propagation-induced errors when PtV is enabled.

Figure 5: Comparison of accuracy and error propagation ratio in HCRE with and without PtV, showing significant reduction in error propagation across hierarchical levels.

Experimental Results

Evaluation on CodRED establishes that HCRE achieves state-of-the-art performance, decisively outperforming both SLM-based and alternative LLM-based hierarchical text classification (HTC) baselines. With LLaMA-3.1-8B-Instruct as backbone, HCRE reaches 45.35 micro F1 and 58.19 binary F1 (closed) and 34.91/49.33 (open), exceeding the RoBERTa+NEPD baseline by 2.39 and 5.52 points in micro F1 and binary F1 in the closed setting.

Ablation studies confirm that each component—hierarchical relation trees, level-wise tree construction, multi-view PtV—makes a critical contribution. Notably, removing PtV induces a drop of over 7 micro F1 points, validating its centrality to robust hierarchical inference.

Figure 6: Comparison of alternative evaluation metrics (micro F1, P@K, binary F1) and dataset size effects, supporting the choice of micro F1 and binary F1 as primary metrics.

Figure 7: Example prompt template as formatted for LLM classification at each node (including path, head/tail entities, and reduced relation options).

HCRE’s performance is robust to tree depth (see Figure 8) and stable across multiple LLM backbones (Qwen2.5, Gemma2), underscoring generality beyond specific architectures or prompt lengths.

Figure 8: Performance vs. tree depth $L$; HCRE is robust, consistently achieving top results across different values.

Practical and Theoretical Implications

By reframing cross-document RE as a multi-level hierarchical classification problem, HCRE demonstrates that LLMs can be successfully adapted to structured multi-class prediction tasks that outstrip the O(N) prompt and context constraints of direct classification. The hierarchical construction is economical (tree construction costs < \$1 via GPT-4o) and adapts across relation schemas and datasets. PtV additionally links hierarchical LLM-based inference to recent advances in verification-augmented prompting and deliberative rollouts, opening the door to more reliable, interpretable predictions, even under severe class imbalance.

Practically, HCRE can be deployed in information extraction pipelines over large-scale knowledge bases (e.g., Wikidata) for mining inter-document relationships. Theoretically, HCRE relates to error-correction coding over hierarchical label spaces and to fine-grained calibration of LLM decision points; it suggests that LLMs can be partitioned into local classifiers with minimal interlevel error transfer when coupled with PtV-like verification steps.

Future Directions

Key limitations include (1) the lack of support for reasoning over multiple text paths per instance (i.e., cross-path inference), and (2) random sampling of suboptimal nodes during training, which could induce bias in the training distribution. Addressing these will further enhance scalability and accuracy. Moreover, future research could investigate joint entity-relation-tree co-evolution, and extending PtV to general purpose multi-level reasoning tasks in LLMs. The hierarchical, verification-augmented paradigm has direct applications in multi-label, taxonomically organized extraction and classification problems far beyond relation extraction.

Conclusion

HCRE provides decisive evidence that hierarchical structuring, coupled with prediction-then-verification, is a critical enabler for LLMs in complex classification tasks with large candidate spaces and non-trivial error propagation. The approach outperforms flat classification, is robust to design choices, leverages scalable, automated tree construction, and opens new avenues in verifiable, interpretable LLM inference for structured information extraction.

---

Reference: "HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy" (2604.07937).