Interpreting synthetic-task performance: reliance on prior knowledge vs reasoning

Ascertain how much large language models rely on prior parametric factual knowledge when solving synthetic unseen tasks and distinguish whether failures arise from insufficient reasoning complexity or from missing background knowledge that models typically exploit.

Background

The authors note that evaluations restricted to synthetic, unseen tasks avoid contamination by prior training data but leave important questions unresolved. Success may demonstrate reasoning in isolation, but does not indicate the typical reliance on prior knowledge; conversely, failures are ambiguous between reasoning difficulty and lack of background knowledge.

SynthWorlds is proposed to control task difficulty and parametric knowledge relevance by constructing parallel corpora and mirrored tasks, allowing the contribution of prior knowledge versus reasoning to be quantified—addressing the ambiguity highlighted in this open question.

References

Fifth, activation versus in-context teaching: because both E1 and the TRUE hint make the rule available in context, we cannot fully separate “activating latent knowledge” from “supplying missing knowledge”; the near-ceiling basic buckets and the sufficiency of a light hint make the activation reading more parsimonious but not decisive, and directly probing whether a model can state the rule unprompted would better separate the two---future work.

— SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics  (2609.11180 - Chen et al., 10 Sep 2026) in Section “Limitations and Conclusion,” fifth limitation

Crucially, evaluations based only on synthetic unseen tasks still leave open questions about performance. Success demonstrates reasoning in isolation, but it does not reveal how much models typically rely on prior knowledge as a scaffold.

— SynthWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Models  (2510.24427 - Gu et al., 28 Oct 2025) in Introduction (Section 1)

We do not have that contrast. Our no-context arm uses a third prompt, neither strict nor permissive but a plain instruction to answer to the best of the model's knowledge, while the wrong-context arm uses the permissive template that invites the model to combine supplied information with what it knows.

— The Answer Path and the Grounding Instruction in LLM Question Answering over Knowledge Graphs  (2609.10237 - Canedo, 9 Sep 2026) in Section 5, “Wrong Context and the Poisoned Retrieval Path,” subsection “The comparison that would isolate suppression, and the one we have”