Binary Tree Search for Data Synthesis

• Binary Tree Search for Data Synthesis is a method that models data spaces as binary trees to systematically partition and explore features for synthetic dataset generation.
• The approach leverages adaptive partitioning and Monte Carlo tree search principles to optimize sample diversity and improve quality using impurity reduction metrics.
• Empirical results demonstrate improvements of up to 10 percentage points in accuracy and a 45% increase in diversity over baseline methods in various benchmarking tasks.

Binary tree search for data synthesis refers to a class of algorithms that utilize binary tree structures—where each non-leaf node branches into exactly two children—to systematically partition and explore data or instruction spaces for the purpose of generating diverse, high-quality synthetic datasets. Two principal instantiations are evident in recent literature: (1) adaptive binary tree search for evolving instructions in LLM alignment (Li et al., 2024), and (2) task-space partitioning for comprehensive and diverse synthetic data generation via TreeSynth (Wang et al., 21 Mar 2025). Both paradigms exploit the hierarchical and recursive nature of binary trees, but differ in their operational principles, target objectives, and utility in data-centric machine learning workflows.

1. Binary Tree Representation in Data Synthesis

The foundational principle is to model the space of possible data (or instructions) as a tree structure, where each node corresponds to a subspace or intermediate object, and branching decisions encode data transformation or partition operations (Wang et al., 21 Mar 2025). In the TreeSynth framework, the full data space $\mathcal{X}$ (e.g., all possible math problems or code snippets for a task) is recursively split at each internal node $v$ by selecting a feature index $j$ and threshold $\tau$:

$$\mathcal{X}_v^{(L)} = \{x \in \mathcal{X}_v : x_j \leq \tau\},\quad \mathcal{X}_v^{(R)} = \{x \in \mathcal{X}_v : x_j > \tau\}.$$

Splits are chosen by maximizing impurity reduction (e.g., variance or entropy decrease) over LLM-generated “pivot” samples local to node $v$. The recursion continues to a fixed depth $d$, resulting in $2^d$ mutually exclusive and exhaustive leaf subspaces (Wang et al., 21 Mar 2025). This ensures that samples synthesized within different leaves are distinct and cover the global space in a nonoverlapping manner.

In contrast, instruction synthesis via tree search (e.g., IDEA-MCTS) models each node as an instruction state $z_t$, which can be evolved by applying one of several prompt-driven actions $a$ (such as “Add Key Constraints”) via an LLM, forming the next child node $z_{t+1} = p_e(z_t, a)$ (Li et al., 2024). The tree thus encodes the evolutionary trajectory of instruction transformations.

2. Binary Tree Search Algorithms for Data and Instruction Synthesis

Data synthesis via binary trees typically proceeds in two phases: (a) constructing the tree by recursive partitioning, and (b) sample generation within each subspace. Pseudocode for TreeSynth is as follows (Wang et al., 21 Mar 2025):

Tree Construction:

1. At each node $v$, generate $l$ pivot samples via LLM.
2. Choose $(j^*, \tau^*)$ to maximize

$$I(\mathcal{P}_v) - \frac{|\mathcal{P}_v^{(L)}|}{l}I(\mathcal{P}_v^{(L)}) - \frac{|\mathcal{P}_v^{(R)}|}{l}I(\mathcal{P}_v^{(R)}),$$

where $I(\cdot)$ measures impurity (e.g., variance).

1. Recurse on $\mathcal{X}_v^{(L)}$ and $\mathcal{X}_v^{(R)}$ until maximum depth.

Data Synthesis:

• For each leaf subspace $\mathcal{X}_i$, generate $n$ data points with a prompt constraining $x \in \mathcal{X}_i$.
• Collect the union of samples from all leaves to form the synthetic dataset.

In instruction synthesis, an episode of Monte Carlo tree search (MCTS) starts from a root instruction and recursively selects, expands, evaluates, and simulates child nodes according to the Upper Confidence Bound (UCT) policy:

$$\mathrm{UCT}(z) = V(z) + C\sqrt{\frac{\ln N(\mathrm{parent}(z))}{N(z)}},$$

where $N(z)$ is the node visit count, $V(z)$ is the value estimate, and $C$ balances exploration/exploitation. Rewards combine predicted quality, diversity, and complexity scores of instructions (Li et al., 2024).

3. Binary Constraint: Effects, Adaptations, and Implementation

Imposing a strict binary branching constraint modifies the search and synthesis processes:

• TreeSynth: Each internal node splits into precisely two child subspaces by a single feature threshold. The depth $d$ determines that there are $N = 2^d$ leaves, with each leaf corresponding to a distinct atomic region of the global data space (Wang et al., 21 Mar 2025). This enables uniform sample allocation to leaves, yielding both coverage and mutual exclusivity.
• MCTS-based Instruction Synthesis: Rather than expanding $n$ (e.g., $5$ in IDEA-MCTS) children per node, each expansion now selects exactly two actions, potentially privileging actions by greedy reward and novelty measures. The UCT selection, rollout logic, and backpropagation remain unchanged, but the search focuses computational resources along two high-value evolution chains. This deepens exploration at the expense of breadth; high-value but low-immediate-reward actions may remain unexpanded. To alleviate lost breadth, the binary restriction can be probabilistically relaxed (Li et al., 2024).

A plausible implication is that this binary modification, while computationally conservative, may find deeper, higher-reward instruction chains, particularly valuable when API calls or LLM inference are resource-constrained.

4. Guarantees and Properties: Diversity, Coverage, and Allocation

A defining property of binary tree partitioning is that leaves are mutually exclusive and collectively exhaustive. Uniform sample allocation per leaf (with total budget $N_\mathrm{tot}$, $n_i = \lceil N_\mathrm{tot}/2^d \rceil$) ensures comprehensive space coverage and avoids concentration within a few modes (subspace collapse) (Wang et al., 21 Mar 2025). Diversity is formally quantified by the cosine dissimilarity metric:

$$D(S) = 1 - \frac{1}{M(M-1)} \sum_{p \neq q} \mathrm{cosine}(x_p, x_q),$$

with $D(S) \in [0,1]$, where larger values imply higher pairwise dissimilarity and thus greater dataset diversity (Wang et al., 21 Mar 2025).

In instruction evolution, direct evaluation of each candidate by composite reward (quality, diversity, complexity) tightly aligns the search process with the desired properties. This contrasts with unguided approaches reliant on temperature sampling or simple iterative editing (Li et al., 2024).

5. Rebalancing and Augmentation of Existing Datasets

Binary search trees facilitate the rebalancing of existing ("real") datasets by mapping each sample $x$ to its unique leaf $\mathcal{X}_{i(x)}$ and assigning sampling weights inversely proportional to the local density $m_i$ of that leaf (Wang et al., 21 Mar 2025):

$$w_i = \frac{1}{m_i}\bigg/ \sum_{j=1}^{N} \frac{1}{m_j},\quad w(x) = w_{i(x)}.$$

This enables the construction of TREESYNTH-balanced variants of standard datasets, correcting class or subspace imbalance and enhancing the representativeness for downstream training.

6. Empirical Results and Comparative Analysis

Empirical investigations on math reasoning (GSM8K, MATH), code generation (HumanEval, MBPP), and psychology (SimpleToM) benchmarks reveal that binary TreeSynth improves downstream LLM accuracy by approximately $10$ percentage points and diversity metrics by up to $45.2\%$ over strong baselines such as temperature sampling, Evol-Instruct, and human-authored datasets (Wang et al., 21 Mar 2025). Illustrative results from (Wang et al., 21 Mar 2025):

Method	GSM8K acc	GSM8K $D$	MATH acc	MATH $D$	HumanEval acc	HumanEval $D$
Temp. Sampling	54.9	0.45	24.3	0.29	45.7	0.32
Evol-Instruct	61.0	0.39	24.6	0.19	49.4	0.25
TreeSynth (binary)	66.7	0.35	30.3	0.12	50.0	0.19

For MCTS-based instruction synthesis, evolving from random iterative refinement to binary-constrained MCTS yields increased mean evaluation scores (quality/diversity/complexity from $2.19$ to $3.81$) and up to $5\%$ accuracy gains on open-domain instruction-following benchmarks in low-resource conditions (Li et al., 2024).

7. Computational Complexity and Scalability

For a binary tree of depth $d$, the total number of nodes is $O(2^d)$. Tree construction requires $O(2^d T_\mathrm{LLM} l)$ steps, where $T_\mathrm{LLM}$ denotes LLM sampling cost and $l$ is the pivot sample count. Data synthesis per leaf with per-leaf sample count $n$ costs $O(2^d T_\mathrm{LLM} n)$. Storage and downstream fine-tuning steps scale linearly with total sample count $N_\mathrm{tot}$ (Wang et al., 21 Mar 2025). For MCTS approaches, constraining branching factor to two limits the sampling budget per expansion and allows deeper search at fixed computational expenditure (Li et al., 2024).

In summary, binary tree search frameworks for data synthesis, as instantiated in TreeSynth and binary-MCTS variants, provide structured, coverage-guaranteed, and diversity-aware sample generation regimes. Empirical evidence demonstrates significant gains over prior art, with the binary constraint offering an efficient tradeoff between search depth and breadth, well-suited for domains where data coverage, balance, and resource efficiency are paramount (Li et al., 2024, Wang et al., 21 Mar 2025).