Synthetic Probing Tasks

• Synthetic probing tasks are evaluation benchmarks built with controlled, synthetic data to isolate specific model properties such as linguistic and algorithmic features.
• They leverage minimal mutations, randomization, and structured data generation alongside diagnostic classifiers to assess if models internalize targeted features.
• Serving as practical tools across NLP, code understanding, and vision, these tasks reveal both diagnostic strengths and limitations in model generalization and transferability.

Synthetic probing tasks are systematically constructed, controlled evaluation datasets and associated objective functions—often classification or regression—that target specific properties or capabilities of neural or symbolic models. Originally developed to dissect the information content of learned representations, these tasks are typically generated through minimal mutations, randomization, or synthetic data generation pipelines. They act as fine-grained diagnostic tools for isolating linguistic, reasoning, structural, or algorithmic features, operating orthogonally to real downstream tasks. Synthetic probes have become central across natural language processing, code understanding, vision, and agentic reasoning as a way to objectively interrogate foundation models regarding their internalization of specific concepts, invariances, or inductive biases.

1. Fundamental Principles and Definitions

Synthetic probing tasks are designed to answer targeted questions such as “Does a representation encode grammatical gender?” or “Can an LLM simulate the steps of an algorithm beyond surface recall?” Unlike naturalistic evaluation benchmarks, which sample real-world distributions, synthetic probing tasks either operate by

• stripping away naturally occurring context to focus on a feature-in-isolation (e.g., type-level word morphology (Şahin et al., 2019)),
• injecting controlled, minimally invasive mutations into existing data (e.g., swapping function words (Kim et al., 2019)), or
• generating data and labels through explicit, fully-parameterized processes (e.g., code execution traces, grid-world transitions (Jin et al., 2024), customized image rendering (Lacoste et al., 2020)).

A canonical setup consists of a dataset $\mathcal{D}$ of instances $x_i$, each labeled $y_i$ with respect to a well-defined property. The probe is a classifier $g_\phi$ trained on fixed representations $h(x_i)$ extracted from a frozen model $f$, whose only goal is to detect the extractability of $y_i$ without further adaptation of $f$. Crucially, synthetic probing often includes “control” tasks where $y_i$ is random or irrelevant, providing a baseline for measuring memorization and representational capacity (Voita et al., 2020, Ravichander et al., 2020).

2. Prototypical Task Classes and Construction Methodologies

There is wide methodological diversity in synthetic probing task design, determined by research goals and the domain under study. Prominent typologies include:

(a) Type-level morphological tasks (NLP)

Şahin et al. (2019) introduced 15 type-level probes—such as Case Marking, Gender, Mood, Number, POS, Person, Tense, Voice, etc.—constructed from UniMorph paradigms and Wikipedia word frequencies, covering 24 morphologically diverse languages. For each task, forms ambiguous for the target feature are removed, positive and negative (including "None") classes are balanced, and train/dev/test splits are maintained strictly non-overlapping (Şahin et al., 2019).

(b) Minimal mutation tasks (linguistic acceptability/NLI)

Kim et al. (2019) defined nine function-word probes by structurally mutating sentences—e.g., swapping wh-words, prepositions, articles—then annotating them for acceptability or entailment class. The controlled mutation operator guarantees that, modulo a single changed word, the data distribution is matched, isolating the target linguistic phenomenon (Kim et al., 2019).

(c) Code and algorithmic reasoning probes

INSPECT introduced a taxonomy of 15 synthetic tasks for code transformers, spanning surface (length), syntactic (identifier class, typo detection), structural (operator/variable count, complexity metrics), and semantic (identifier/keyword replacement) properties (Karmakar et al., 2023). La Malfa et al. proposed code-simulation tasks—automatically generated programs designed to isolate reasoning primitives such as sequential execution, branching, parallel state, and recursion depth (Malfa et al., 5 Feb 2025).

(d) Synthetic image generation and control

Synbols generates images from a fully specified directed graphical model over symbol identity, font, texture, geometry, and occlusion; downstream classification, robustness, and meta-learning tasks are instantiated by varying the generative factors and dataset splits (Lacoste et al., 2020).

(e) Agentic, tool-using, and compositional reasoning

TaskCraft creates challenging multi-hop, multi-tool tasks by recursively extending atomic tool-using queries, then merging or compounding them to scale complexity. Atomic, depth-extended, and width-extended tasks are automatically verified for correctness via tool-execution and LLM-judging (Shi et al., 11 Jun 2025).

3. Evaluation Protocols, Metrics, and Probing Architectures

Synthetic probing universally employs “diagnostic” classifiers: probes are simple (often linear or shallow MLPs) and are trained exclusively on frozen representations. Key components:

• Input: pre-extracted feature vectors (embedding, encoder output, [CLS] token, or pooled hidden state).
• Objective: cross-entropy for classification, MSE for regression, sometimes structured losses (ranking, binary mediation).
• Data: strictly balanced and split to avoid leakage; controlled negative sampling (None-class), data augmentation, and anti-memorization splits are commonly used.
• Metrics: accuracy, F1, area under ROC, mutual information lower-bounds, or (for code/data) token-sequence exact-match, MAE, Levenshtein distance.
• Advanced: Minimum description length (MDL) metrics combine prediction accuracy and probe complexity (parametric or data-codelength), directly penalizing memorization (Voita et al., 2020).

To relate probe performance to real utility, some studies report rank correlations (Spearman’s $\rho$) between probe accuracy and downstream scores for tasks such as POS-tagging, dependency parsing, NER, XNLI, or QA (Şahin et al., 2019, Karmakar et al., 2023).

4. Empirical Insights, Failure Modes, and Interpretational Limits

Empirical use of synthetic probes has revealed both their utility and intrinsic limitations:

• Diagnostic value: Well-constructed probes reliably indicate the presence or absence of targeted features (e.g., morphological richness, type or structural code attributes).
• Model class dependence: Contextualized word embeddings (D-ELMo) and structurally informed code models (GraphCodeBERT) far outperform static or baseline models on most probes (Şahin et al., 2019, Karmakar et al., 2023).
• Surface vs. depth: Superficially robust probe accuracy is often possible even for synthetic or random properties, especially with large representations; thus, probe accuracy can overestimate task relevance unless causal, capacity-controlled, or information-theoretic calibrations are applied (Ravichander et al., 2020, Voita et al., 2020).
• Failure on distribution shift: Transferability is often poor; synthetic tasks may not correlate strongly with downstream task success unless the probe explicitly targets features relevant to those applications (Mohiuddin et al., 2020, Kirch et al., 21 Nov 2025).
• Lazy pattern recognition and overfitting: LLMs often display “shortcut” behavior on synthetic code/execution tasks, e.g., outputting sorted lists without simulating steps, thus failing on adversarial variations or when presented with semantically redundant information (Malfa et al., 5 Feb 2025).
• Data domain effects: Use of synthetic/off-policy data for probe training can yield high performance only when the synthetic distribution matches the inference domain, with cross-domain gaps remaining substantial (Kirch et al., 21 Nov 2025).

5. Advanced Methodologies: Causal and Information-Theoretic Probing

Limitations of accuracy-based probes have motivated new synthetic control strategies:

• Randomized controls: Formulating synthetic tasks with random or decoupled labels quantifies how much “incidental encoding” exists; models often recover these labels well above chance if capacity permits (Ravichander et al., 2020).
• MDL-based probing: The minimum description length framework reframes probe evaluation as a data compression problem, balancing the bits required to encode labels given representations and the cost of encoding the probe model. Genuine structure allows small codes, whereas synthetic/random tasks incur high codelengths, even when accuracy gaps are minimal (Voita et al., 2020).
• Causal mediation and latent structure: Jin et al. demonstrated SCM-based synthetic task generation (e.g., grid-world programs) where causal interventions allow direct measurement of whether LMs encode true latent variables, distinguishing “representation” from “probe extraction” (Jin et al., 2024).

6. Best Practices and Future Directions

Designing, interpreting, and extending synthetic probing tasks demands attention to rigor, validation, and practical relevance:

• Construction: Ensure tasks test for a single property and that data splits are meticulously controlled for leakage, overlap, and confounds.
• Baselines and controls: Always include matched random or decoy tasks to calibrate probe accuracy against memorization.
• Interpretation: Prefer information-theoretic or causal mediation metrics over raw accuracy in high-capacity settings. Use cross-domain testing and rank correlations to relate probes to extrinsic utility.
• Extension: Scale synthetic task suites to new languages, domains (NLP, code, vision), model architectures, and compositional or hierarchical phenomena.
• Benchmark unification: Construct libraries and task generators allowing community benchmarking of algorithmic, tool-use, structural, and linguistic skills under controlled synthetic regimes (Malfa et al., 5 Feb 2025, Shi et al., 11 Jun 2025).

Synthetic probing tasks remain a vital cornerstone of model analysis, providing scalable, controlled, and highly diagnostic lenses for uncovering what models learn, where they generalize, and which properties are incidentally versus causally encoded. Their limitations under standard accuracy metrics necessitate continual methodological refinement, especially as models and target tasks increase in scale and complexity.