LAMBADA: Backward Chaining for Automated Reasoning in Natural Language

Abstract: Remarkable progress has been made on automated reasoning with natural text, by using LLMs (LMs) and methods such as Chain-of-Thought and Selection-Inference. These techniques search for proofs in the forward direction from axioms to the conclusion, which suffers from a combinatorial explosion of the search space, and thus high failure rates for problems requiring longer chains of reasoning. The classical automated reasoning literature has shown that reasoning in the backward direction (i.e. from the intended conclusion to supporting axioms) is significantly more efficient at proof-finding. Importing this intuition into the LM setting, we develop a Backward Chaining algorithm, called LAMBADA, that decomposes reasoning into four sub-modules. These sub-modules are simply implemented by few-shot prompted LM inference. We show that LAMBADA achieves sizable accuracy boosts over state-of-the-art forward reasoning methods on challenging logical reasoning datasets, particularly when deep and accurate proof chains are required.

• The paper introduces LAMBADA, a modular pipeline of few-shot language-model calls that uses backward chaining for fact checking, rule selection, goal decomposition, and sign agreement.
• The method improves depth-five accuracy over Chain-of-Thought by 44% on ProofWriter-PUD, 37% on PrOntoQA, and 43% on naturalistic ParaRules, while using up to 11.8 times fewer calls than Selection-Inference.
• The paper shows that CoT can achieve correct labels through invalid proof shortcuts, whereas LAMBADA produces more reliable reasoning traces, though it remains costly and limited primarily to modus ponens and entailment classification.

LAMBADA (LLM augmented BAckwarD chAining) addresses a persistent weakness of LLM (LM) based deductive reasoning: the reliance on forward-chaining proof search, which suffers from a combinatorially expanding search space as reasoning depth grows. The paper, authored by Kazemi, Kim, Bhatia, Xu, and Ramachandran of Google Research, imports the classical automated-reasoning insight that backward, goal-directed proof search is more efficient than forward inference from axioms, and implements it as a modular pipeline of few-shot prompted LM calls over natural language theories (2212.13894).

Motivation and positioning

The authors motivate the work with two observations. First, scaling provides limited benefit on logical tasks relative to other language tasks; they cite evidence that the Gopher family of LMs shows substantially worse scaling returns for logic-based tasks. Second, finetuning LMs on logical reasoning datasets tends to induce spurious correlations (e.g., between rule counts and labels) rather than genuine reasoning ability, and prompting methods such as Chain-of-Thought (CoT) struggle with proof planning on multi-hop problems. Existing LM reasoning systems, notably ProofWriter and Selection-Inference (SI), embed elements of forward chaining (FC), which requires selecting a subset of facts and rules from the theory — a combinatorial search — and lacks a natural halting criterion for declaring failure to prove. Backward chaining (BC), by contrast, decomposes the goal recursively into sub-goals, requires no subset selection, and has a natural halting criterion in the maximum search depth.

The LAMBADA method

LAMBADA operates on theories consisting of natural language facts and rules of the form "If P then Q," and a goal $G$ to be proved, disproved, or declared Unknown (open-world assumption). The algorithm is a depth-first search over the theory driven by four LM modules, each implemented purely with in-context demonstrations:

• Fact Check verifies whether a fact entails or contradicts $G$, implemented as a two-step fact-selection plus verification sub-procedure (retried twice, removing the selected fact on failure).
• Rule Selection identifies rules whose consequent unifies with $G$, factored into a one-time consequent-extraction step and a goal-conditioned applicability step.
• Goal Decomposition breaks $G$ into sub-goals given a unifying rule's antecedent.
• Sign Agreement determines whether the sign (polarity) of the goal agrees with the sign of the rule consequent, deciding Proved versus Disproved.

A reranking heuristic orders applicable rules from shortest to longest, on the intuition that shorter rules have fewer antecedents and higher success probability. Caching and exact-match cycle detection reduce redundant LM calls and prevent infinite loops. The authors note that the cycle check only matches goals exactly, so semantically equivalent sub-goals (e.g., a sub-goal that is the negation of the root) are not pruned.

Label accuracy results

Experiments use PaLM 540B with temperature-zero decoding, evaluated on ProofWriter (OWA), PrOntoQA (fictional characters version), and ParaRules (crowdworker-paraphrased ProofWriter), with depths up to 5 hops. LAMBADA is compared against CoT and SI. The headline results are substantial:

Benchmark (Depth-5)	Relative improvement of LAMBADA
ProofWriter-PUD (includes Unknown labels)	+44% over CoT, +56% over SI
PrOntoQA	+37% over CoT, +113% over SI
ParaRules (naturalistic)	+43% over CoT

The ParaRules result is particularly significant because the theories are written in naturalistic, non-templatic language — the setting where combining LM text understanding with symbolic search structure is most valuable. SI was excluded from ParaRules due to poor performance and high call counts. On ProofWriter-PD (Proved/Disproved only), CoT remains competitive and its accuracy does not degrade substantially with depth — an anomaly the paper investigates next.

Proof accuracy and spurious shortcuts

A central finding is that CoT's high label accuracy on ProofWriter-PD is largely spurious. Manual verification of 50 Depth-5 examples where CoT predicted correctly showed that only 28% had valid proof chains; hallucination of facts and rules accounted for 48% of errors, and hallucinated content frequently created shortcuts to the correct label. The appendix quantifies the underlying dataset bias: in 9.2% of multi-hop examples the goal matches a rule consequent, and 98.9% of those are labeled Proved; the presence of "not" in a goal is strongly predictive of the Disproved label. Because LAMBADA's modules never condition on the label, they are impervious to such input–label correlations. This is a strong claim with a caveat: it rests on manual verification of a 50-example sample, and the spurious-correlation analysis is specific to ProofWriter-PD.

Why forward chaining degrades

The paper provides mechanistic evidence for FC's failure modes. On PrOntoQA Depth-5, the success rate of SI's $k$-th inference decreases with $k$, consistent with the hypothesis that the growing theory makes the selection search progressively harder — contradicting the intuition that later inferences should be easier as the distance to the goal shrinks. SI also produces redundant inferences: among failed Depth-5 ProofWriter-PUD examples, only 29% of five-inference runs contained no duplicates, with 7% producing a single unique inference five times. SI additionally over-predicts Disproved (binary) and Unknown (three-way), sometimes below the majority-class baseline. These analyses support the paper's claim that backward chaining, requiring only linear scans over facts and rules, is structurally better suited to LM-based modular reasoning.

Is backward CoT sufficient?

A natural alternative is simply prompting CoT with backward-written proofs. The paper shows this does not suffice: backward CoT achieves label accuracy comparable to forward CoT, but produces substantially lower-quality proofs on Depth-5 ProofWriter-PD. The authors argue this indicates backward CoT may lean more heavily on spurious correlations in the PD setting, and conclude that the modular decomposition in LAMBADA — not merely the direction of the chain — is essential. This is consistent with earlier findings on finetuned transformers by Gontier et al. The authors concede that LAMBADA's traces could in principle be used to finetune smaller models (e.g., via STaR-style bootstrapping), which they leave unexplored.

Module analysis and the role of scale

Isolated module evaluation on ProofWriter validation data shows Rule Selection is the weakest module, followed by Goal Decomposition; Fact Check reaches 0.94 accuracy with single-fact selection and near-perfect accuracy with two facts; Sign Agreement is near-perfect. Scaling behavior is informative: at PaLM 62B, Goal Decomposition and Sign Agreement hold up while Fact Check and Rule Selection degrade sharply; at 8B, all modules approach random performance. The authors attribute this to the one-to-many comparison structure of the selection-type modules, and argue that the granularity of problem decomposition should be matched to base model scale — a reasonable but untested prescription.

Efficiency and robustness

LAMBADA is considerably more query-efficient than SI: 3.8× fewer LM calls at Depth-1 and 11.8× fewer at Depth-5 of ProofWriter-PUD, though it still requires many more calls than single-pass CoT. Lexical robustness experiments — replacing entity names, animals, adjectives, and verbs with novel tokens, and substituting rule templates with novel phrasings — show performance remains essentially unchanged, and LAMBADA on modified test sets still outperforms baselines on the original sets. A practical caveat: the authors manually corrected quality issues in the first 500 ParaRules test examples (altered antecedents, introduced antecedents, over-specific rules, pronoun substitutions), so the ParaRules numbers reflect a cleaned subset.

Limitations

The paper explicitly enumerates constraints on applicability. LAMBADA addresses only entailment-style classification, not open-ended question answering over theories. It assumes all rules are provided in the prompt and does not handle implicit knowledge drawn from the LM's parameters. Reasoning is restricted to modus ponens; other inference rules (proof by contradiction, disjunction elimination) are unaddressed. The sequential, dependent structure of LM calls precludes batching, increasing latency. And despite efficiency gains over SI, the call count remains much higher than CoT, raising cost.

Conclusion

LAMBADA demonstrates that a classical backward-chaining proof procedure, with each inference step delegated to few-shot prompted LM modules, yields sizable gains in both label accuracy and proof validity over forward-chaining and end-to-end CoT approaches on deductive reasoning benchmarks, with the largest gains at depth 5 and on naturalistic text. The paper's most consequential contribution is the empirical case that goal-directed search, rather than merely better prompting, is the appropriate backbone for LM-based multi-hop reasoning. Open questions include extension to non-modus-ponens inference, integration of parametric knowledge, batched execution, and whether verifier-based reranking of LAMBADA proofs (left as future work) would compound the reported gains.