LinkSyn: Knowledge Graph Synthesis Framework

• LinkSyn is a controlled synthesis framework that leverages KP graphs and diffusion-based generation to create diverse QA data for improved LLM training.
• It integrates KP extraction, weighted graph construction, and hybrid graph-walk sampling to balance coverage and popularity while controlling difficulty levels.
• Empirical results demonstrate significant improvements on benchmarks such as MMLU and CMMLU, establishing LinkSyn as a promising approach for knowledge-intensive LLM pre-training.

LinkSyn is a knowledge point (KP) graph-based synthesis framework introduced in the context of LLM training data generation. It is designed to synthesize diverse question-answering (QA) data from multiple seeds that are strongly linked by knowledge points and sampled from graph walks, while allowing flexible control over discipline and difficulty distributions and balancing KP coverage and popularity. In the reported implementation, executing LinkSyn produces LinkQA, a multi-disciplinary synthetic QA corpus with 50B tokens, and continual pre-training with that corpus yields substantial gains on knowledge-intensive benchmarks (Zhang et al., 2 Aug 2025).

1. Problem formulation and conceptual scope

LinkSyn is motivated by a specific bottleneck in LLM development: the scarcity of high-quality, diverse training data. Its central premise is that QA synthesis should be organized not around isolated seed instances, but around structured relations among the knowledge points extracted from those instances. Each QA example is represented as

$D_i=(t_i,s_i,h_i,K_i),$

where $t_i$ is the question text, $s_i$ is the discipline label, $h_i$ is the difficulty level, and $K_i=\{k_{i1},\dots,k_{i,n_i}\}$ is the set of extracted knowledge points. This representation makes the knowledge structure explicit and supports subsequent graph-based sampling and controlled synthesis (Zhang et al., 2 Aug 2025).

A common simplification is to treat LinkSyn as a generic graph sampler for data augmentation. That characterization is incomplete. The framework couples KP extraction, weighted graph construction, path sampling under a coverage–popularity trade-off, multi-seed conditional generation, answer refinement, and explicit difficulty adjustment. The synthesis target is therefore not merely more data, but data whose knowledge composition, disciplinary distribution, and difficulty profile can be manipulated within a unified pipeline.

2. Knowledge point extraction and graph construction

The first stage of LinkSyn is KP extraction from large-scale seed QA data. The extraction pipeline proceeds in three steps: DeepSeek-R1 generates initial KP annotations on 20 M seed QAs; these annotations are distilled into a small, high-confidence subset; and Qwen2.5-14B-Instruct is then fine-tuned to serve as the KP extractor. After deduplication through a two-stage procedure consisting of edit-distance clustering and co-occurrence-vector clustering, the process yields approximately 10 M consolidated, high-quality KPs (Zhang et al., 2 Aug 2025).

The extracted KPs define a weighted, undirected graph

$G=(K,E,W,\Phi),$

where $K$ is the set of unique knowledge points, $E\subseteq K\times K$ contains an edge $(k_p,k_q)$ iff there exists some QA $D_i$ with both $t_i$0 and $t_i$1 in $t_i$2, the edge-weight function is

$t_i$3

and the incidence map

$t_i$4

returns all QA instances containing $t_i$5 (Zhang et al., 2 Aug 2025).

This formulation makes co-occurrence counts the primary structural signal. In effect, LinkSyn uses observed KP co-membership within seed QAs as the basis for deciding which concepts should be traversed together during synthesis. A plausible implication is that the graph operationalizes local conceptual relatedness in a manner that is directly usable for seed grouping, without requiring external ontologies or manually curated taxonomies.

3. Coverage–popularity trade-off and the knowledge value function

A defining feature of LinkSyn is the explicit balancing of two competing sampling objectives. The first is “Coverage,” which encourages rare KPs; the second is “Popularity,” which matches empirical KP frequencies in the seed data. These are encoded through two reference distributions: $t_i$6 for the uniform distribution, and

$t_i$7

for the empirical distribution (Zhang et al., 2 Aug 2025).

Given a candidate sampling distribution $t_i$8, LinkSyn defines the knowledge value function as

$t_i$9

where $s_i$0 can be squared-Euclidean or KL divergence. Under either divergence, the unique minimizer is

$s_i$1

This gives a closed-form interpolation between uniform exploration and empirical fidelity (Zhang et al., 2 Aug 2025).

The significance of this construction is methodological. Rather than relying on heuristic over-sampling or ad hoc reweighting, LinkSyn formalizes distribution control as an optimization problem with an explicit optimum. This makes the coverage–popularity compromise analyzable and tunable through $s_i$2, and it ties the synthesis regime to a well-specified target distribution over KPs.

4. Graph-walk sampling and seed-group formation

LinkSyn builds KP paths of length $s_i$3 using two random-walk policies and then mixes them. In the coverage-priority walk, the start node is sampled as $s_i$4, and transitions are uniform over neighbors: $s_i$5 In the popularity-priority walk, the start node is sampled as $s_i$6, and transitions are weighted by edge co-occurrence counts: $s_i$7 After independently sampling $s_i$8 paths under each policy, the framework constructs a hybrid path distribution

$s_i$9

This hybridization is the operational mechanism through which the earlier value-function intuition is realized in path sampling (Zhang et al., 2 Aug 2025).

For each path $h_i$0 in $h_i$1, LinkSyn samples a target difficulty $h_i$2 and discipline $h_i$3. It then selects, for each KP on the path, a seed QA from $h_i$4 whose $h_i$5 best matches the sampled attributes, using

$h_i$6

The selected instances are collected into a seed group $h_i$7 (Zhang et al., 2 Aug 2025).

This stage clarifies that LinkSyn does not synthesize from a single exemplar. Its immediate conditioning object is a seed set assembled from multiple QAs aligned to a graph path. The framework therefore treats generation as composition over linked knowledge fragments rather than local variation around one prompt.

5. Diffusion-based synthesis and answer refinement

Given a sampled seed group $h_i$8, LinkSyn uses DeepSeek-R1 as a diffusion-based conditional generator. The conditioning input is the concatenation or interleaving of the $h_i$9 seed QAs. The generator diffuses through a pretrained LLM-based noise schedule, optimizes a denoising objective, and produces a batch of diverse target QAs that preserve logical links among the original KPs on the path. The reported objective is

$K_i=\{k_{i1},\dots,k_{i,n_i}\}$0

where $K_i=\{k_{i1},\dots,k_{i,n_i}\}$1 is the ground-truth QA, $K_i=\{k_{i1},\dots,k_{i,n_i}\}$2 is the noisy version, and $K_i=\{k_{i1},\dots,k_{i,n_i}\}$3 is conditioned on $K_i=\{k_{i1},\dots,k_{i,n_i}\}$4 (Zhang et al., 2 Aug 2025).

Generated outputs are subsequently passed through DeepSeek-V3 for final answer polishing and format conformance. In the algorithmic overview, this post-generation stage is followed by cleaning via benchmark-contam filters before inclusion in the final corpus (Zhang et al., 2 Aug 2025).

An important clarification follows from this design. LinkSyn is not only a graph-based retrieval mechanism, nor only a generation prompt-construction scheme. Its synthesis stage is explicitly diffusion-based and multi-seed conditioned, with answer refinement separated from the core generation step. This modular decomposition distinguishes structural seed selection from textual realization.

6. Difficulty control, corpus construction, and empirical results

Difficulty adjustment is a built-in control mechanism rather than a secondary annotation layer. LinkSyn samples target difficulty from a user-specified distribution $K_i=\{k_{i1},\dots,k_{i,n_i}\}$5; the paper gives as an example

$K_i=\{k_{i1},\dots,k_{i,n_i}\}$6

over H1–H5. By selecting support instances that minimize difficulty mismatch while preferring discipline matches, the framework mixes in more difficult seeds as needed. Empirically, this raises the fraction of high-difficulty questions in LinkQA by approximately $K_i=\{k_{i1},\dots,k_{i,n_i}\}$7 relative to baselines (Zhang et al., 2 Aug 2025).

The algorithmic overview specifies the following inputs: seed QA set $K_i=\{k_{i1},\dots,k_{i,n_i}\}$8, KP graph $K_i=\{k_{i1},\dots,k_{i,n_i}\}$9, path-length set $G=(K,E,W,\Phi),$0, mixing parameter $G=(K,E,W,\Phi),$1, attribute distributions $G=(K,E,W,\Phi),$2, and target total paths $G=(K,E,W,\Phi),$3. The main steps are KP extraction and graph construction, coverage- and popularity-based path sampling with hybridization, target attribute sampling and seed selection, diffusion synthesis via DeepSeek-R1, answer refinement with DeepSeek-V3, and contamination-aware cleaning. The output is LinkQA, a 50 B token synthetic QA corpus (Zhang et al., 2 Aug 2025).

The reported pre-training experiment uses Llama-3 8B, continuing from 2 T tokens with an additional 40 B tokens blended 1:1 with KnowEdu and LinkQA. The optimizer is Adam, with learning rate $G=(K,E,W,\Phi),$4 linearly decayed, batch size 960, sequence length 8 K, and bfloat16 precision. On MMLU and CMMLU, LinkQA yields an average $G=(K,E,W,\Phi),$5 absolute improvement over the pre-training baseline; across 12 tasks, the overall average gain is $G=(K,E,W,\Phi),$6. Consistent improvements are also reported for 1.7 B and 16 B model sizes, and from 2 T to 10 T initial FLOPs checkpoints. The paper characterizes these results as establishing new SOTA results (Zhang et al., 2 Aug 2025).

Taken together, these results position LinkSyn as a controlled synthesis framework whose distinctive contribution lies in combining KP-graph-guided path sampling, value-function balancing of coverage and popularity, diffusion-based multi-seed generation, and difficulty-aware instance selection. This suggests that its primary importance is not any single component in isolation, but the integration of structural sampling and conditional generation into a single data-construction regime for knowledge-intensive LLM training.