--- title: Synthetic Pre-training on SCMs url: https://www.emergentmind.com/topics/synthetic-pre-training-on-scms type: topic --- # Synthetic Pre-training on SCMs Synthetic pre-training on Structural Causal Models (SCMs) refers to a framework for continued pretraining of large language models (LLMs) using a diversified, vast corpus of tabular prediction tasks synthesized directly from randomly generated SCMs. This approach, as operationalized in MachineLearningLM, aims to enhance the in-context learning (ICL) capability of LLMs—particularly their ability to learn from many-shot tabular classification demonstrations—while explicitly preserving the underlying model's general knowledge and reasoning faculties. The central methodology serializes millions of artificial tabular classification problems into token-efficient prompts, leverages decision-level distillation from tree-based models such as random forests (RFs), and utilizes standard next-token language modeling objectives in pretraining. Empirically, this yields strong, monotonic scaling of accuracy with the number of in-context examples, outperforming existing LLM baselines and matching the performance of robust classical ML models across several domains. ## 1. Structural Causal Model Formalism and Task Generation In synthetic pre-training, each data-generating process for a tabular task is formalized as a structural causal model: \[ \mathcal{S} = (U,\;V,\;F,\;P_U) \] where $U = \{\varepsilon_v: v \in V\}$ denotes exogenous noise variables, $V = \{x_v: v = 1, \dots, |V|\}$ endogenous variables (features), $F=\{f_v : \Pa(v) \to x_v\}_{v \in V}$ a collection of node-specific structural functions, and $P_U$ the product distribution over $U$, which is typically $\prod_v \mathcal{N}(0,1)$. SCMs are instantiated via: - Sampling a directed acyclic graph (DAG) with a layer structure akin to a fully-connected multilayer perceptron (depth $L$, width $w$). - Imposing edges between all nodes in adjacent layers, with a random topological ordering. - Defining $f_v$ as a randomly chosen nonlinear activation (e.g., $\tanh$, ReLU, SiLU, $\sin$, $\exp$) with 70% probability, or a gradient-boosted tree regressor (GBR) fit to Gaussian noise (30% probability). - Exogenous noise terms $\varepsilon_v$ drawn i.i.d. from $\mathcal{N}(0,1)$ (alternatives such as uniform or Student’s t supported but unimportant empirically). Sampling from such an SCM involves a topological pass through the DAG: \[ x_v = f_v(x_{\Pa(v)}) + \varepsilon_v, \qquad \varepsilon_v \sim \mathcal{N}(0,1) \] This framework yields richly structured and statistically diverse synthetic tabular datasets. ## 2. Construction of Synthetic Tabular Prediction Tasks Once an SCM is sampled, tabular classification tasks are constructed as follows: 1. Generate a large set of i.i.d. samples $\{x^{(i)}\}_{i=1}^n$ by ancestral sampling on the SCM. 2. Designate one coordinate $y^\star = x_{v^\star}$ as a latent regression "score." 3. Select a classification arity $K \in \{2, \dots, 10\}$ with quantile boundaries $\tau_1 < \dots < \tau_{K-1}$ defined on $y^\star$’s empirical distribution. 4. Discretize $y^\star$ into labels $y$ by assigning class indices according to quantile bins, randomly shuffling class IDs. \[ y = \argmax_{k \in \{1, \dots, K\}} \left[ \mathbb{1}\{\tau_{k-1} \leq y^\star < \tau_k\} \right] \] 5. Subsample $M$ "demonstration" $(x, y)$ pairs ($M \leq 1024$), $N=50$ query $(x)$ inputs, and set dimensionality $d \in [5, 50]$. This yields millions of tasks with varied type, dimensionality, class distribution, and underlying generative dependencies. ## 3. Token-Efficient Serialization and Prompt Construction To maximize context utilization, MachineLearningLM employs a serialization strategy based on comma-delimited tabular rows without natural language filler. Each synthetic prompt consists of: - An instruction header $H$. - Demonstration set $S$ containing $M$ rows: each is $\texttt{demoID}_j:$ followed by $d$ feature integers and a label. - Query block $Q$ with $N$ rows: each is $\texttt{queryID}_j:$ followed by $d$ feature integers. All features are z-normalized and quantized into $[0,999]$: \[ i = \mathrm{clip}(\mathrm{round}(120\,z + 500), 0, 999) \] The output target is a single JSON array, with predicted labels for all queries: \[ \text{[}\{\texttt{"id": …, "label": …}\}, \dots \text{]} \] This design delivers a 3x–6x increase in task-per-context window efficiency and enables up to 50x throughput improvement via batch inference. ## 4. Teacher-Guided Distillation and Pretraining Objective To stabilize early-stage learning on pure synthetic distributions, the framework employs a random-forest "warm-start" distillation process: - Each synthetic classification task is first evaluated by a small random forest (RF) trained on the demonstration set. - Tasks where the RF fails to exceed a conservative label-distribution baseline ($p_0 = \max(\sum_k p_k^2, \max_k p_k)$) are discarded, verified with a binomial test ($\alpha=0.2$), Cohen’s $\kappa>0$, and balanced-accuracy checks. - During the warm-up ($\sim$1 million tasks), only those query instances where $\hat{y}_\text{RF} = y_\text{true}$ are retained. The LLM is trained to reproduce RF labels using the standard left-to-right next-token negative log-likelihood: \[ \mathcal{L}(\theta) = -\sum_{t=1}^{T} \log p_\theta(y_t \mid \mathbf{x}, y_{