---
title: 'RefineX: Scalable Data Refinement for LLMs'
url: https://www.emergentmind.com/topics/refinex
type: topic
---

# RefineX: Scalable Data Refinement for LLMs

RefineX is a programmatic data refinement framework for large-scale pre-training corpora, designed to systematically improve the quality of data prior to training large language models (LLMs). Unlike traditional rule-based document-level filtering or neural generative refinement, RefineX implements efficient, fine-grained, character-level deletion programs distilled from expert-guided end-to-end edits. This methodology enables scalable, precise, and reliable removal of irrelevant or low-quality spans, while preserving the diversity and naturalness of the pre-training distribution and minimizing the introduction of artifacts.

## 1. Formalization and Scope

Let $X = \{t_1, \dots, t_N\}$ be a raw text corpus, where each $t_i$ is a character sequence. The objective is to construct a refinement operator $R$ such that, when applied instance-wise ($D_R = R(X)$), the downstream performance $Perf(\theta; T)$ of an LLM with parameters $\theta$—trained only on $D_R$, under a fixed compute budget—is maximized over a task set $T$. The search for $R$ is restricted to character-level, deletion-only actions:
\[
R^* = \argmax_{R \in \mathcal{R}} \max_{\theta: \mathrm{Pretrain}(\theta; R(X))} Perf(\theta; T)
\]
The admissible $R$ comprises only deletion programs, excluding insertions and substitutions, to ensure operational minimality and avoid style drift or information “hallucination” [2507.03253].

## 2. Expert-Guided Distillation Pipeline

The refinement process consists of a two-stage pipeline:

### a. Seed Corpus Sampling

- Raw data $X$ is scored by the DataMan method, assigning a quality-level (1–5) to each document.
- Approximately 5 million documents are sampled, maintaining the DataMan stratification.
- Each long document is split into overlapping 12,000-character chunks to fit the expert LLM input window.

### b. Expert End-to-End Deletion

- The Qwen2.5–72B–Instruct LLM is prompted to refine, with explicit instructions for deletion-only editing and retention of on-topic links.
- Decoding hyperparameters: top-$p=0.8$, top-$k=20$.
- Outputs include a refined version $t^E$ of each input $t$ and an enumerated “modification_reason”.
- The resulting $(t, t^E)$ pairs form a high-quality but expensive and potentially over-edited demonstration set.

### c. Minimal Edit Distillation

- For each $(t, t^E)$, the minimal edit distance is computed via Levenshtein algorithms (Python `difflib.get_opcodes`), yielding $\mathcal{O}_{\min}$, a sequence of deletions, insertions, and replacements realising $E(\mathcal{O}_{\min}, t) = t^E$.
- All non-deletion ops are dropped, and only deletions $\mathcal{O}_{\mathrm{del}} = \{d_1, \dots, d_K\}$ (each contiguous span) are retained.
- Each $d_j$ is mapped to a domain-specific language (DSL) statement:
  1. `remove_lines(start_line, end_line)`
  2. `remove_str(line, del_str)`
  3. `keep_all()` (no modification)
- Filtering removes: (i) cases with insertion/replace spans $\geq$ 20 chars, (ii) trivial examples ($<$10 chars deleted).
- The finalized distillation set comprises about two million high-fidelity $(t, \mathcal{F})$ pairs.

## 3. Model Architecture and Training

The program generator is a decoder-only Transformer initialized as Qwen-3-Base (0.6B params), featuring:

- 24 layers, hidden size 1,536, 24 self-attention heads.
- Vocabulary: 32k BPE.
- Context window: $\leq$12k characters (approx 2k tokens).

The model input is of the form:

```
[DOC] ...raw chunk text... [ENDDOC] RefineX program:
```
Output is a linearized DSL sequence, for example: 
```
remove_lines(3,5); remove_str(8,"http://spam"); keep_all();
```

Training objective is cross-entropy over the gold program tokens:
\[
\mathcal{L}(\theta) = -\sum_{n=1}^N \sum_{t=1}^{L_n} \log P_\theta (y_t^{(n)} | y_{<t}^{(n)}, x^{(n)})
\]
Regularization is achieved by layer dropout of 0.1, output length cap at 512, and a program length penalty. Scaling to 1.7B, 4B, 8B models offered marginal improvements, but with higher latency.

## 4. Corpus-Scale Inference and Guarantees

Corpus-wide refinement is executed with the following pseudocode:

```
Algorithm RefineX_Inference
Input: Raw corpus X, chunk size W_char, RefineX model M_θ
Output: Refined corpus R(X)
Initialize R(X) = []
for each document D in X in parallel:
   C = ChunkSplit(D, W_char)
   refined_chunks = []
   for chunk c ∈ C:
     program = M_θ.generate(c)    # beam=1, max_len=512
     text' = ExecuteProgram(program, c)
     refined_chunks.append(text')
   R(X).append(MergeChunks(refined_chunks))
return R(X)
```

This process is linear in total character count and per-chunk inference time. Large batch sizes (~8 chunks/GPU) are accommodated with vLLM for throughput, and overlapping chunks prevent context loss.

Only deletion operations are introduced, preserving the original token and n-gram statistics, natural distribution, and multimodal evidence in the data.

## 5. Experimental Protocol and Comparative Evaluation

### a. Datasets and Baseline Methods

- Base corpus: RedPajama-V2 (400B tokens pre-filter), subsampled/refined to 20B tokens per method.
- Methods:
  1. Raw (unfiltered)
  2. Rule-based (Gopher, C4, FineWeb, Comb)
  3. LLM-based filter (Prox-D)
  4. Programmatic refine (ProX-C)
  5. RefineX

### b. Pretraining Details

- LLM architectures: Llama-2 family at 350M and 750M scales.
- Training regime: 10k steps, batch 1,024×2,048 tokens, total 20B tokens.
- Hyperparameters: LR=$5\times10^{-4}$ (cosine decay), AdamW, weight decay 0.1, 500-step warmup.

### c. Evaluation Protocol

- LightEval suite: ARC-C, ARC-E, CSQA, HellaSwag, MMLU (57 sub-tasks), OBQA, PIQA, SIQA, WinoGrande, SciQ.
- Each accuracy measured zero-shot on 1,000 samples/task.

## 6. Quantitative Profile

Table: Downstream Accuracy (750M LLM, 20B Tokens)

| Dataset Filter   | Avg. Accuracy | Δ vs. Raw |
|------------------|---------------|-----------|
| Raw              | 41.6%         | –         |
| ProX-C           | 42.4%         | +0.8%     |
| RefineX          | 42.9%         | +1.3%     |

Further, for the Comb baseline:
- Comb: 42.1%, +ProX-C 42.4%, +RefineX 42.8%.
- With Prox-D: 42.0% → +ProX-C 43.5% → +RefineX 44.7%.

Token efficiency: models pretrained on 10B RefineX-refined tokens match—or slightly exceed—the performance of 20B Comb-filtered tokens.

Instance-level quality (DataMan evaluation): For score=3 sub-corpus, RefineX improves 41.2% of documents (degrades 4.58%), with average post-refinement score 3.45. E2E refinement improves 59.0% but has higher risk of over-editing (degrades 3.39%).

Over-editing risk: Number of new words per 1,000 refined tokens — E2E 15.06, ProX-C 0.17, RefineX 0.00 for low-quality (score=2) spans, demonstrating the deletion-only constraint is effective in limiting artifact introduction.

## 7. Comparative Merits, Limitations, and Outlook

**Reliability:** The staged distillation system—expert E2E to minimal deletion program—safeguards against model hallucinations and style drift. Supervision remains highly aligned with expert instructions, benefiting reproducibility and generalization.

**Precision:** Deletion-only edit programs serve as a conservative, reliable filter, minimizing spurious modifications and preserving original linguistic and stylistic features.

**Efficiency:** Both the DSL program (few tokens per instance) and a low-latency backbone reduce compute and memory overhead during inference.

**Coverage:** Character- and line-granularity permits more surgical, targeted curation, surpassing document-level and coarse programmatic approaches in fine control.

Limitations: The pipeline requires expensive large-scale expert LLM inference (Qwen2.5–72B), incurring substantial GPU-hour cost (~12k GPU-hrs). The scope is presently limited to deletions; carefully introduced substitution or metadata edits could further enhance quality but may risk style drift or information loss.

*This suggests* a future extension could incorporate proprietary expert annotators (e.g., GPT-4) for the distillation phase, and revisit carefully bounded non-deletion edit operations to balance data quality with distribution fidelity [2507.03253].

Source: https://www.emergentmind.com/topics/refinex