Hierarchical Node Compression (HNC)

• Hierarchical Node Compression (HNC) is a data augmentation strategy that merges consecutive parent–child nodes in MCTS trees to diversify reasoning data for LLM training.
• It uses controlled merging to compress complex reasoning steps, transferring key metrics like MC-Scores while drastically reducing computational demands.
• Empirical results on datasets such as PRM800K, GSM8K, and MATH500 show that HNC boosts reward model accuracy and stability with minimal overhead.

Hierarchical Node Compression (HNC) is a data augmentation strategy for tree-structured reasoning processes, developed to enhance the stability, diversity, and robustness of reward models in LLM training. By compressing segments of Monte Carlo Tree Search (MCTS)-generated reasoning trees through controlled merging of parent–child nodes, HNC amplifies the variety of reasoning step sequences, injects controlled label noise, and incurs minimal computational overhead in large-scale automated reasoning pipelines (Wang et al., 16 Mar 2025).

1. Motivation and Conceptual Foundations

The development of HNC originates from the need to efficiently create high-diversity, robust reasoning data for training Process Reward Models (PRMs) and, more generally, for Hierarchical Reward Models (HRMs) in LLMs. MCTS is commonly used to annotate reasoning trajectories by recursively simulating possible step sequences; however, the computational cost becomes prohibitive with deep, wide trees required for stable MC-Score estimation, with a reported budget of ∼2,457 A100-GPU hours for standard datasets. This computational bottleneck restricts tree expansion and limits the diversity of reasoning patterns present in the data.

HNC addresses this limitation by randomly merging consecutive reasoning steps within the existing MCTS trees, thus producing compressed trees that yield both finer- and coarser-grained reasoning examples. This augmentation is computationally lightweight (∼30 minutes on a single A100 GPU), enables broader coverage of reasoning sequence types, and introduces mild stochastic perturbations that are empirically shown to enhance model generalization and stability (Wang et al., 16 Mar 2025).

2. Formal Definition and Mathematical Operations

Let $T = (V, E)$ denote an MCTS tree, where each node $v \in V$ represents a partial chain-of-thought step, and edges $E$ define parent–child relationships. Each node $v$ is associated with a text descriptor $\mathrm{text}(v)$ and an MC-Score $S(v)$, defined as the normalized count of correct leaves underneath $v$: $$S(v) = \frac{\#\,\{\text{correct leaves under }v\}}{\#\,\{\text{all leaves under }v\}}$$ HNC selects a subset of eligible edges $(u \rightarrow v)$ and constructs a new node $w$ via:

• Text merging:

$v \in V$0

• Score transfer: $v \in V$1.
• Tree rewiring:
  • $v \in V$2 replaces $v \in V$3’s link from its parent, if any
  • $v \in V$4 inherits $v \in V$5’s children as its own

The resulting tree $v \in V$6 contains new compressed nodes, with the overall depth and step granularity reduced along merged branches. Only a controlled fraction of parent–child pairs are merged to avoid collapsing the entire structure, thus preserving essential tree diversity.

3. Algorithmic Implementation

The canonical algorithm for HNC is as follows:

$E$8 In practice, $v \in V$7 is selected to produce sufficient variability without excessive structural collapse.

4. Computational Complexity and Efficiency

The cost of generating the initial MCTS trees scales exponentially with tree depth and branching factor. In contrast, HNC’s augmentation pass iterates over each edge once, with only $v \in V$8 operations per edge (merging, redirection, deletion/insertion). Thus, the overall complexity is $v \in V$9, with trivial GPU/memory consumption relative to MCTS tree expansion. The time footprint is empirically stated as $E$030 minutes on a single A100 (80GB), compared to thousands of GPU-hours needed by MCTS for a single dataset (Wang et al., 16 Mar 2025).

5. A Step-by-Step Example of HNC on an MCTS Tree

Consider an MCTS mini-tree with the following structure:

$E$9

By merging Step 1 → Step 2b, the tree under that branch becomes: $v$0 The merged branch becomes one level shallower, while the MC-Score is inherited from Step 2b.

6. Integration into the Hierarchical Reward Model (HRM) Training Pipeline

The HNC-augmented trees play a dual role in HRM pipeline construction:

1. Large sets of raw MCTS trees $E$1 are generated per reasoning task.
2. PRMs are first trained on basic stepwise (fine-grained) pairs from these trees.
3. HNC is then applied to each $E$2 to generate compressed variants $E$3 containing parent–child merges.
4. Both fine-grained (original) and coarse-grained (merged) pairs are extracted for HRM training, with labels derived from PRM or MC-Scores.
5. HRM is ultimately trained on this union, improving its ability to evaluate both individual and multi-step reasoning coherence.

This strategy ensures exposure to both diversity in short steps and robustness to variable step granularity, reflecting real-world reasoning trajectories (Wang et al., 16 Mar 2025).

7. Empirical Evaluation and Measured Impact

Empirical results reported on PRM800K, GSM8K, and MATH500 datasets indicate that HRMs trained with HNC augmentation outperform baseline PRMs in both absolute accuracy and output stability. Using the Qwen2.5-7B-Math-Instruct policy under Best-of-$E$4 sampling, HRMs achieve a score of 0.655 at $E$5 versus PRM’s 0.600, with improved stability for all $E$6 up to 64.

• Generalization to new reasoning domains (e.g., GSM8K, Math500) is also strengthened, particularly for more challenging splits.
• The augmentation process itself is negligible in computational demands, requiring less than 1/100th the resources of raw MCTS expansion.

A plausible implication is that HNC facilitates more robust training by increasing data diversity and introducing mild label noise, which improves tolerance to overfitting and enhances model performance under distributional shift (Wang et al., 16 Mar 2025).

---

Summary Table: Key Operational Dimensions of HNC

Aspect	Requirement/Result	Reference
Core operation	Merge parent–child nodes in MCTS tree	(Wang et al., 16 Mar 2025)
Computational cost	$E$7; minutes on single GPU	(Wang et al., 16 Mar 2025)
Training impact	+5.5 points Best-of-8 accuracy (PRM800K dataset)	(Wang et al., 16 Mar 2025)
Usage context	Augmentation for Hierarchical Reward Models	(Wang et al., 16 Mar 2025)

Hierarchical Node Compression demonstrably improves the data efficiency and reliability of reward modeling for LLM-driven reasoning, with practical benefits in both empirical stability and generalization.