MachineLearningLM: In-Context Tabular Learning

• MachineLearningLM is a continued-pretraining framework that extends LLM in-context learning for tabular prediction by leveraging millions of synthetic supervised tasks.
• It employs token-efficient serialization and random-forest teacher distillation to optimize performance and achieve systematic many-shot generalization across domains.
• This approach improves classification accuracy by up to 15 percentage points over baseline LLMs while preserving core natural language understanding and reasoning abilities.

MachineLearningLM is a continued-pretraining framework designed to enable a general-purpose LLM to achieve robust in-context machine learning (ICL) capabilities for tabular prediction tasks, while retaining standard natural language understanding and reasoning. Distinct from conventional fine-tuning for each downstream task, MachineLearningLM introduces a portable pretraining procedure based on millions of synthetic supervised learning problems, scaled up to 1,024 in-context examples per prompt and leveraging causal structure and model distillation. This approach delivers systematic, monotonic many-shot generalization in ICL classification and strong empirical performance in comparison to both prior LLMs and classical tabular learning methods.

1. Motivation and Problem Formulation

MachineLearningLM addresses the persistent gap between the theoretical promise of LLMs' ICL and their empirical performance on standard ML tasks, specifically many-shot tabular classification. Off-the-shelf LLMs exhibit broad world knowledge and reasoning but are ineffective at extracting reliable statistical patterns from sets of in-context examples without gradient descent or task-specific fine-tuning. The central goal is to enable LLMs to amortize ML task solving across many novel, unseen structured prediction problems, including those with domain shift and substantial numbers of demonstrations, without requiring further model updates.

The learning setting is formalized as:

• Given a prompt containing $k$ in-context examples $(x_1, y_1), \ldots, (x_k, y_k)$, and a set of test queries $x_{k+1}, \ldots, x_{k+m}$, the LLM must output predictions $y_{k+1}, \ldots, y_{k+m}$.
• The tasks are diverse tabular classification problems, with features generated according to a structural causal model (SCM).
• The LLM is trained to predict outputs in a strict, deterministic JSON format for end-to-end parsing.

2. Pretraining Pipeline: Synthetic Supervised Task Generation

The pretraining corpus is synthesized at large scale by sampling prediction tasks from millions of structural causal models (SCMs). For each task:

• A random DAG with $v$ nodes defines the causal relations. Each variable $x_v$ is sampled as $x_v = f_v(\text{pa}(v), \epsilon_v)$, where $f_v$ is a function over parent variables $\text{pa}(v)$ and local noise $\epsilon_v$.
• Downstream tabular classification tasks are instantiated by selecting target variables and generating independently sampled rows (via the SCM rules), yielding both demonstration and evaluation sets.
• Feature normalization is performed in two steps:
  • Z-score normalization: $z = (x - \mu) / \sigma$ for each numeric feature.
  • Discretization to $i = \text{clip}(\text{round}(120z + 500), 0, 999)$, so all values lie in $0$–$999$ and map to a single tokenizer token, maximizing prompt compression.

A compact comma-delimited format serializes examples (with a premade JSON output expectation), improving context-window efficiency by a factor of 3–6× versus naive prompting.

Random-Forest Teacher Distillation

As a warm-up, a random-forest classifier (RF) is trained on the full synthetic pool for each task and used as a teacher for the LLM. Only demonstration/test pairs where the RF prediction matches the ground truth are included. This step encourages the LLM to internalize tree-based reasoning heuristics, supports stable curriculum progression, and increases early signal-to-noise, especially for highly imbalanced or low-signal tasks.

After warm-up/warm-start distillation, the LLM is pretrained to optimize left-to-right next-token log-likelihood on the JSON-structured outputs:

$$\mathcal{L}(\theta) = -\sum_{t} \log p_{\theta}(y_t \mid x, y_1, \ldots, y_{t-1})$$

covering task formats, constraints, and a fixed row ordering for output reproducibility.

3. Many-Shot Scaling Law and Performance Across Domains

MachineLearningLM exhibits a pronounced "many-shot scaling law": empirical accuracy increases monotonically as the number of in-context demonstrations $k$ rises from 8 up to 1,024 for tabular classification. This stands out in contrast to baseline LLMs, which typically plateau or degrade with increasing context examples due to ineffective numerical modeling.

When evaluated on out-of-distribution tabular classification tasks drawn from finance, physics, biology, and healthcare, MachineLearningLM (Qwen-2.5-7B-Instruct with LoRA rank 8) exceeds strong LLM baselines such as GPT-5-mini by approximately 15 percentage points on average. In regimes with hundreds of demonstrations, it achieves random-forest-level accuracy "without any task-specific training," showing robust generalization on tasks far from the SCMs observed during pretraining.

On standard LLM benchmarks (e.g., MMLU), MachineLearningLM maintains chat and reasoning ability despite large-scale domain-specific continued pretraining, achieving 75.4% accuracy in a 50-shot setting and strong numeracy on subdomains such as statistics and physics.

4. Token Efficiency, Prompt Engineering, and Batch Prediction

A central bottleneck for many-shot ICL is prompt length and computational cost. MachineLearningLM introduces three technical solutions:

• Token-efficient tabular encoding: Demonstrations are masked and serialized as comma-delimited rows, with numbers mapped to the $[0,999]$ bin and all features optimized for minimum token count.
• Multi-query batch prediction: At inference, multiple test queries are included per prompt, reusing the shared demonstration block and yielding up to $50 \times$ improvement in amortized forward-pass throughput.
• Order-robust confidence aggregation: To counteract prompt order sensitivity, the model predicts each query across $V=5$ randomly shuffled variants of the demonstration set. The next-token probabilities are summed:

$\tilde{p}(y_j) = \sum_{i=0}^{V-1} p_i(y_j)$

and the output argmax is selected. This confidence-aware self-consistency increases robustness in label prediction for many-shot cases.

5. Robustness and Preservation of General Capabilities

Unlike many domain-adaptive pretraining regimes, MachineLearningLM's approach leaves the base model's architecture and tokenizer untouched. As a result, general natural language understanding, factual recall, and reasoning remain intact. There is no observed regression on general chat benchmarks or reasoning workflows. This positions MachineLearningLM as a hybrid ICL/assistant model suitable for both structured ML prediction and broad interactive use.

6. Limitations and Future Research

MachineLearningLM achieves strong ICL on tabular classification, but continuous regression and unsupervised learning remain open challenges. The design is specific to token-efficient prompting, and while the model generalizes across many tabular domains, scaling to more complex output spaces or regression targets requires new techniques. Order sensitivity is partially addressed by self-consistency over prompt permutations, but further gains may require algorithmic advancements or permutation-invariant architectures.

Potential directions include extending the method to richer structured outputs, integrating additional classical teacher models, leveraging real-world tabular datasets in pretraining, and exploring direct incorporation of inference-time calibration or ensembling.

7. Summary Table: Technical Innovations and Empirical Results

Aspect	Approach	Result/Insight
Training Data	Millions of synthetic SCM tabular tasks, 8–1,024 shots	Systematic scaling of ICL; OOD domain coverage
Pretraining Regime	Random-forest teacher warm-up, autoregressive LM on output	Teacher stabilization, robust early learning
Prompt Engineering	Token-efficient tabular serialization, [0,999]-norm	3–6× more examples per prompt, 50× throughput
Inference Robustness	Self-consistency by permutation aggregation (V=5)	Order-robust likelihood-based prediction
Classification Generalization	Outperforms GPT-5-mini by ~15 pp, RF-level accuracy	Many-shot scaling law (better with more demos)
Preservation of Chat/Reasoning	No drop on MMLU, multi-domain knowledge	Maintains general LLM capabilities post-adaptation

This portable continued-pretraining paradigm demonstrates that, with judicious task synthesis, teacher-guided distillation, token-efficient serialization, and robust aggregation, a general-purpose LLM can acquire strong in-context classification ability—directly via prompting—on a diverse set of tabular machine learning tasks, all while retaining its foundational natural language faculties (Dong et al., 8 Sep 2025).