FAC Synthesis for LLM Post-Training

• FAC Synthesis is a diversity-driven framework that generates synthetic data by targeting missing, task-relevant activation features in LLMs.
• It employs Feature Activation Coverage (FAC) computed from sparse autoencoder features to identify and fill gaps in data representation.
• Empirical results show enhanced performance in toxicity detection, reward modeling, behavior steering, and instruction following with fewer samples.

Searching arXiv for the named framework and closely related work to ground the article in current literature. FAC Synthesis is a diversity-driven data synthesis framework for LLM post-training that defines diversity in an interpretable internal feature space rather than in text space. In "Less is Enough: Synthesizing Diverse Data in Feature Space of LLMs" (Li et al., 11 Feb 2026), the framework is built around Feature Activation Coverage (FAC), a metric over sparse-autoencoder features extracted from LLM activations. The method first identifies task-relevant features that are present in an anchor corpus but absent from a seed dataset, then generates synthetic samples that explicitly express those missing features. The reported motivation is data-centric optimization of LLMs for settings such as instruction following, toxicity detection, reward modeling, and behavior steering, where synthetic data quality and diversity directly affect downstream performance (Li et al., 11 Feb 2026).

1. Problem setting and conceptual scope

FAC Synthesis addresses a specific limitation in post-training data construction: many synthetic-data pipelines increase apparent variety in surface form while failing to cover task-relevant internal model features. The framework is motivated by the observation that post-training data is often limited, long-tailed, and expensive to collect, while synthetic data pipelines such as Alpaca, Self-Instruct, and MAGPIE can produce near-duplicates, biased coverage, and missed rare but important behaviors (Li et al., 11 Feb 2026).

The central claim is that common diversity metrics are poorly aligned with downstream utility. Surface-level metrics such as Distinct-$n$, $n$-gram entropy, POS-tag diversity, and sentence length, as well as embedding-level metrics such as cosine distance, semantic entropy, embedding covariance, and clustering metrics, measure variation in text or embeddings but do not directly measure how the target model internally represents data. The paper therefore treats diversity as a property of model activations, specifically of sparse, interpretable latent features recovered from hidden states (Li et al., 11 Feb 2026).

This reframing matters because FAC Synthesis is not merely a prompting heuristic. Its objective is to expose the model to missing, task-relevant activation patterns. A common misconception is to read it as another synthetic-instruction pipeline whose novelty lies in prompt engineering. The paper’s position is narrower and more technical: the novelty is the use of an SAE-derived feature basis, a coverage metric defined on that basis, and an overview loop that targets uncovered features directly (Li et al., 11 Feb 2026).

2. Feature Activation Coverage and the SAE feature space

The framework begins with a sparse autoencoder trained on token-level activations from a chosen LLM layer. Given an activation $x \in \mathbb{R}^d$, the encoder produces a sparse feature vector

$$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$

with tied-weight reconstruction

$\hat{x} = z W_{\text{SAE}}.$

The classical SAE objective is

$$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$

In practice, the paper uses a Top-$K$ SAE: for each token, only the $K$ largest activations are retained, with Top-$K = 20$ per token. Sequence-level features are obtained by max-pooling across tokens after a template prefix cutoff $t_0$,

$n$0

yielding a fixed-dimensional feature activation vector $n$1 (Li et al., 11 Feb 2026).

The SAE is trained on approximately $n$2K diverse prompts and about $n$3M tokens, using hidden activations from LLaMA-3.1-8B layer 16. The feature dimension $n$4 is set by a scaling law $n$5 with $n$6, and training uses AdamW with learning rate $n$7 for $n$8 epochs (Li et al., 11 Feb 2026).

Task relevance is then imposed on this feature basis. For each feature, the authors collect top-activating spans and use GPT-4o-mini to summarize what the spans have in common and classify the feature as Yes, Probably, Maybe, or No for task relevance. Human audit on $n$9 features per task reportedly confirms that about $x \in \mathbb{R}^d$0–$x \in \mathbb{R}^d$1 of GPT-4o-selected features are indeed relevant (Li et al., 11 Feb 2026).

FAC itself is defined over a task-relevant feature subset $x \in \mathbb{R}^d$2. For a threshold $x \in \mathbb{R}^d$3, a feature is considered active on an input $x \in \mathbb{R}^d$4 if

$x \in \mathbb{R}^d$5

Given a distribution $x \in \mathbb{R}^d$6, the set of task-relevant features that appear at all is

$x \in \mathbb{R}^d$7

Using an anchor set $x \in \mathbb{R}^d$8 as a proxy for the target domain and a synthetic or seed dataset $x \in \mathbb{R}^d$9, the paper estimates

$$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$0

and defines

$$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$1

The missing-feature set is

$$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$2

Operationally, FAC is therefore the fraction of task-relevant SAE features activated at least once by the synthetic dataset relative to the anchor corpus (Li et al., 11 Feb 2026).

3. FAC Synthesis workflow

The synthesis pipeline has three stages: build the feature space, identify missing features, and generate synthetic data that explicitly covers them. The first stage fixes a base model and layer, trains the SAE, interprets features, and selects the task-relevant subset $$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$3. The second stage compares the anchor corpus and the current seed dataset to compute $$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$4. The third stage synthesizes data per missing feature (Li et al., 11 Feb 2026).

The generation stage is explicitly two-step. For each missing feature $$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$5, the system first constructs a contrastive pair. A natural-language description $$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$6 of the feature is inserted into a feature-aware prompt template $$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$7 and a generator model $$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$8 samples $$z = \sigma(x W_{\text{SAE}}), \quad \sigma = \text{ReLU},$$9 candidates,

$\hat{x} = z W_{\text{SAE}}.$0

Each candidate is scored by the target feature activation $\hat{x} = z W_{\text{SAE}}.$1. The highest-activating sample becomes a positive exemplar $\hat{x} = z W_{\text{SAE}}.$2 and the lowest-activating sample becomes a negative or weak exemplar $\hat{x} = z W_{\text{SAE}}.$3. This yields a feature-specific contrastive anchor pair (Li et al., 11 Feb 2026).

The second step uses that pair as in-context guidance. A contrastive synthesis prompt $\hat{x} = z W_{\text{SAE}}.$4 conditions the generator to produce samples that behave like the positive exemplar. If $\hat{x} = z W_{\text{SAE}}.$5 denotes the sampled candidate set, the retained subset is filtered by the activation threshold:

$\hat{x} = z W_{\text{SAE}}.$6

The final synthetic dataset is

$\hat{x} = z W_{\text{SAE}}.$7

This design is intended to ensure that samples do not merely discuss a feature description but actually activate the corresponding SAE feature strongly enough to count toward coverage (Li et al., 11 Feb 2026).

Several practical details are fixed in the reported implementation. The default generator is LLaMA-3.1-8B-Instruct. A temperature grid $\hat{x} = z W_{\text{SAE}}.$8 with top-$\hat{x} = z W_{\text{SAE}}.$9 is explored, with best results reported around $$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$0. The activation threshold $$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$1 is explored over $$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$2; moderate thresholds in $$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$3 reportedly stabilize the number of missing features and improve AUPRC by filtering weak activations, whereas $$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$4 makes the missing set too small and degrades performance (Li et al., 11 Feb 2026).

A recurrent empirical point is that only a few samples per feature are needed. The paper varies the number of synthetic samples per missing feature from $$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$5 to $$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$6 and reports that downstream performance increases as more samples are added, but that the data-efficiency score, defined as AUPRC divided by $$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$7, decreases. This suggests that most of the gain comes from covering more distinct missing features rather than from scaling the number of samples per feature (Li et al., 11 Feb 2026).

4. Theoretical rationale

FAC Synthesis is justified by two linked arguments: one about the distribution gap between real and synthetic data, and one about sampling error within the synthetic distribution. Let $$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$8 denote the true task distribution, $$L_{\text{SAE}} = \|x - \hat{x}\|_2^2 + \lambda \|z\|_1.$$9 the synthetic distribution, $K$0 an i.i.d. synthetic sample, $K$1 the post-trained model, and $K$2 a bounded loss with $K$3. The paper states the generalization bound

$K$4

The first term is the distribution gap, and the second term is sampling error under the synthetic distribution (Li et al., 11 Feb 2026).

The SAE feature space enters through an upper bound on the total-variation gap. If $K$5 is the SAE feature vector and $K$6 are the feature distributions under $K$7 and $K$8, the paper states

$K$9

The stated implication is that aligning feature distributions reduces a principled surrogate of the domain gap. FAC then operationalizes one coarse but interpretable aspect of that alignment: support coverage over task-relevant features (Li et al., 11 Feb 2026).

The missing-feature argument is strengthened through a support-based surrogate. The paper constructs feature-space distributions $K$0 and $K$1 uniform over feature supports $K$2 and $K$3, and reports that if $K$4, then $K$5. Covering missing features makes that surrogate KL finite and decreasing. This does not show that support coverage is sufficient for full distributional matching, but it does show that missing-feature elimination addresses a concrete feature-space mismatch (Li et al., 11 Feb 2026).

The second theoretical ingredient concerns the entropy of synthetic data generation. The paper gives a PAC-Bayesian argument in which the sampling error depends on the mutual information between the synthetic dataset and model weights,

$K$6

Since $K$7, the two-step synthesis procedure is motivated as a way to reduce generation entropy relative to naive prompting by conditioning on a high-activating positive exemplar and filtering with the SAE (Li et al., 11 Feb 2026).

5. Empirical behavior and downstream performance

The framework is evaluated on four tasks: toxicity detection, reward modeling, behavior steering, and instruction following. In toxicity detection, the setup fine-tunes LLaMA-3.1-8B on HH-RLHF plus synthetic contrastive examples and evaluates on ToxicChat with AUPRC. In reward modeling, a Bradley-Terry-style reward model is trained on the HH-RLHF helpful subset plus synthetic preference pairs and evaluated on RewardBench. In behavior steering, the paper uses Contrastive Activation Addition datasets for sycophancy and survival instinct, reporting Robust Accuracy and Steering Control Rates. In instruction following, Meta-Llama-3-8B is LoRA fine-tuned on synthetic instruction data and evaluated on AlpacaEval 2 with win rate and length-controlled win rate (Li et al., 11 Feb 2026).

The headline results are summarized below.

Task	Reported comparison	FAC Synthesis result
Toxicity detection	Best baseline: SynAlign $K$8 AUPRC	$K$9 AUPRC
Reward modeling	Best baseline: CoT-Self-Instruct $K = 20$0 avg accuracy	$K = 20$1 avg accuracy
Behavior steering	Best baseline SCRs $K = 20$2–$K = 20$3	$K = 20$4 sycophancy, $K = 20$5 survival
Instruction following	Comparable frontier point: MAGPIE with $K = 20$6K synthetic examples	WR $K = 20$7, LC $K = 20$8 with $K = 20$9K synthetic examples

These results are accompanied by a data-efficiency claim: for instruction following, the paper reports performance comparable to MAGPIE while using $t_0$0 less data (Li et al., 11 Feb 2026).

The paper also emphasizes correlation rather than only absolute task scores. Across tasks, FAC is reported to correlate strongly with downstream performance. Table 6 reports Pearson $t_0$1 and Spearman $t_0$2 between FAC and AUPRC for toxicity detection; for reward modeling, $t_0$3 and $t_0$4; for behavior steering and instruction following, Pearson values are reported around $t_0$5–$t_0$6 and Spearman values around $t_0$7–$t_0$8. By contrast, lexical and embedding diversity metrics reportedly show weak or negative correlations with downstream performance (Li et al., 11 Feb 2026).

Ablations reinforce the same interpretation. When the proportion of missing features covered is varied at fixed total sample count or fixed samples per feature, performance increases monotonically with coverage. Increasing dataset size without increasing feature coverage yields only small gains. Likewise, the two-step synthesis strategy produces higher FAC than a one-step description-only prompt for the same sample budget and threshold, and leads to better downstream metrics across all tasks. The paper therefore argues that the decisive variable is distinct task-relevant feature coverage rather than raw sample count (Li et al., 11 Feb 2026).

6. Cross-model transfer, limitations, and interpretive boundaries

One of the paper’s broader claims is that the SAE feature space is partially shared across model families. Using features extracted from LLaMA-3.1-8B-Instruct, the authors generate a single synthetic dataset and fine-tune three different backbones on the toxicity task: LLaMA-3.1-8B-Instruct improves from $t_0$9 to $n$00, Mistral-7B-Instruct from $n$01 to $n$02, and Qwen2-7B-Instruct from $n$03 to $n$04 (Li et al., 11 Feb 2026). A more systematic $n$05 experiment varying feature source, generator, and backbone further supports the view that some task-relevant SAE features are transferable across LLaMA, Mistral, and Qwen.

The paper’s interpretation is that the feature space captures model-agnostic latent concepts such as “drug procurement queries” or “cheating on tests,” and that these can guide data generation for multiple backbones. At the same time, the transfer is not symmetric. The generator matters: LLaMA-3.1-8B-Instruct reportedly yields the best synthetic data even when the target backbone is different. The feature source also matters: for some targets, LLaMA-sourced features outperform in-family features, while for others Mistral-sourced features can be superior (Li et al., 11 Feb 2026).

The limitations are explicit. Training a large SAE on mid-layer activations is computationally nontrivial, though one-time; the reported run uses $n$06 H100 for about $n$07 hours. FAC is defined relative to a specific model layer, feature extractor, and anchor set, so poor representations or an unrepresentative anchor corpus can distort what counts as “missing.” Feature interpretation is imperfect: some features are syntactic or unclear, and GPT-4o-based labeling leaves about $n$08–$n$09 uncertain. The authors also note that complex reasoning may depend on circuits distributed across layers, whereas the reported method uses a single-layer SAE and explicitly states that capturing “sophisticated reasoning features” remains challenging (Li et al., 11 Feb 2026).

The ethical scope is also delimited. Feature-targeted synthesis could be used to amplify harmful behaviors. The paper states that this risk is mitigated by focusing experiments on safety-improving tasks, by filtering and review, and by limiting direct release of potentially harmful synthetic text. A plausible implication is that FAC Synthesis is best understood as a general data-construction methodology whose effects depend on the feature subset selected and on the downstream supervision protocol, rather than as an intrinsically beneficial or harmful synthesis mechanism (Li et al., 11 Feb 2026).