IRZ-CoT: Residual Connection Chain-of-Thought

• The paper introduces IRZ-CoT, a prompting strategy that embeds explicit residual connections into chain-of-thought reasoning, enabling restoration of non-local dependencies.
• IRZ-CoT reconstructs the underlying reasoning graph by restating earlier sub-answers, which helps LLMs resolve complex, multi-step problems more reliably.
• Empirical results show that IRZ-CoT boosts performance by 6–12 percentage points, with gains up to 21.1% on tasks requiring five or more reasoning steps.

IRZ-CoT (Residual Connection Chain-of-Thought) is a prompting strategy for LLMs that augments conventional chain-of-thought (CoT) prompting with explicit residual connections to encode complex, graph-structured reasoning dependencies in multi-step problem solving. Unlike standard linear CoT, IRZ-CoT reconstructs the full reasoning graph by explicitly restating earlier intermediate results at later steps where dependencies exist, enabling LLMs to resolve long-range reasoning dependencies with significantly improved fidelity, especially in tasks requiring five or more sequential inferences (Jiang et al., 2023).

1. Theoretical Motivation and Conceptual Framework

Standard CoT prompting (“Let’s think step by step…”) directs LLMs towards generating reasoning chains as linear sequences, where each step depends exclusively on the immediately prior step. However, in genuine multi-step problems, the underlying logical structure is better represented as a directed acyclic graph (DAG) $G=(V,E)$, where each node $v_i\in V$ represents an intermediate conclusion or sub-answer, and each edge $(v_i\rightarrow v_j)\in E$ denotes a dependency of sub-step $j$ on result $i$. Linear CoT discards all dependencies where $j-i>1$, omitting critical long-range logical links.

IRZ-CoT restores these non-local dependencies by explicitly restating the token span of each required prior sub-answer $v_i$ in the prompt context that precedes the dependent reasoning step $v_j$. These explicit contextual insertions are termed residual connections by analogy to deep residual networks, and their primary role is to facilitate information flow “across” the linear stepwise prompt structure (Jiang et al., 2023).

2. Formalization and Construction of Residual Connections

Given a reasoning problem decomposed into $N$ steps $V = \{v_1,\ldots,v_N\}$ with a dependency graph $v_i\in V$0, IRZ-CoT identifies the set of “missing” residual links $v_i\in V$1—those dependencies omitted by the purely sequential CoT flow. For every $v_i\in V$2, the exact token span of sub-answer $v_i\in V$3 is reintroduced immediately in the prompt for step $v_i\in V$4.

• Standard CoT template:

$j$0

• IRZ-CoT template with residuals:

$j$1

This technique effectively “residualizes” the reasoning trace, aligning the prompt with the true, typically non-linear, logical structure of the solution (Jiang et al., 2023).

3. Prompt Construction Algorithm

The construction of IRZ-CoT prompts proceeds as follows:

1. Parsing: Decompose the problem statement $v_i\in V$5 into a linear CoT outline and extract the full reasoning dependency graph $v_i\in V$6.
2. Residual Link Identification: Detect all residual links $v_i\in V$7 corresponding to non-adjacent reasoning dependencies.
3. Template Augmentation: Embed necessary prerequisite sub-answers from $v_i\in V$8 with exact token repetition at each corresponding reasoning step.
4. Prompt Assembly: Concatenate few-shot exemplars and the residual-augmented chain for the target problem.

Pseudocode outline: $j$2 (Jiang et al., 2023)

4. Empirical Evaluation and Performance Impact

Performance benchmarks across six datasets—including GSM8K, AQUA-RAT, MathQA, SCONE, and StrategyQA—demonstrate substantial gains for IRZ-CoT relative to derived linear CoT. On LLaMA-65B and LLaMA2-70B, IRZ-CoT improves overall reasoning accuracy by 6–12 percentage points, with the most pronounced effect (+14.3% to +21.1%) in problems requiring at least five reasoning steps. The following table summarizes the reported results:

Model	GSM8K	AQUA-RAT	MathQA	SCONE	StrategyQA
CoT (65B)	47.1	33.5	33.0	13.1	70.2
IRZ-CoT (65B)	58.4	42.5	34.1	15.1	70.2
CoT (70B)	52.7	38.1	38.1	23.3	73.1
IRZ-CoT (70B)	65.3	44.4	39.2	24.3	73.1

For problems necessitating at least five reasoning steps:

	65B	70B
CoT	35.0	40.8
IRZ-CoT	56.1	55.1

(Jiang et al., 2023)

5. Prompt Design, Ablations, and Best Practices

Extensive ablation studies address both the optimal placement and form of residuals. Coverage of all reasoning steps with their required prerequisites yields the best accuracy, with particularly strong gains when residuals emphasize long-range dependencies in later steps. Empirical evidence indicates that exact token repetition of prior sub-answers outperforms the use of symbolic variables (by 2–4 percentage points on GSM8K and AQUA), suggesting that LLMs resolve semantic coreference more robustly when given verbatim phrases than arbitrary symbols. Uniform application of residuals (every other step) and especially full coverage are most effective.

Placement	GSM8K@65B	AQUA@65B
No residual	47.1	33.5
First half only	54.5	31.8
Second half only	55.4	34.6
Uniform (every oth.)	58.4	35.8
Full coverage	58.4	42.5

Best practices include: covering every dependent step with its required prerequisites; prioritizing long-range connections; exact token repetition; and using sufficiently large models (≥65B parameters) to fully leverage the implicit graph-structured context (Jiang et al., 2023).

6. Illustrative Example and Prompt Templates

A representative example, adapted from GSM8K, elucidates the contrast between linear and residualized CoT:

Linear CoT:

• Step 1: Pre-purchase total = \$v_i\in V$915 = \$110
• Step 2: From allowance = 3×\$(v_i\rightarrow v_j)\in E$015</li> <li>Step 3: From mowing = 4×\$(v_i\rightarrow v_j)\in E$160</li> <li>Step 4: Remaining = \$(v_i\rightarrow v_j)\in E$260 = \$(v_i\rightarrow v_j)\in E$350/\$7 ≈ 7.14 → 7 driveways

IRZ-CoT (with residuals):

• Step 1: Pre-purchase total = \$(v_i\rightarrow v_j)\in E$415 = \$110
• Step 2: From allowance = 3×\$(v_i\rightarrow v_j)\in E$515</li> <li>Step 3: From mowing = 4×\$(v_i\rightarrow v_j)\in E$660</li> <li>Step 4 (prereq: “\$(v_i\rightarrow v_j)\in E$760” from Step 3): Remaining = \$(v_i\rightarrow v_j)\in E$860 = \$(v_i\rightarrow v_j)\in E$950/\$7 = 7 driveways

Here, Step 4 repeats the exact tokens previously introduced, explicitly marking the cross-step dependency (Jiang et al., 2023).

7. Implications, Limitations, and Future Directions

IRZ-CoT aligns the in-prompt structure with the actual underlying logical reasoning graph, enabling LLMs to solve complex multi-step problems with markedly increased reliability and reduced error rates compared to conventional sequential CoT prompting. The effect is especially pronounced for problems with deep reasoning chains, where vanilla CoT fails to surface or maintain relevant intermediate facts.

A plausible implication is that as problem complexity—and thus the density of non-local dependencies—increases, explicit in-context alignment via residual connections will remain indispensable until LLMs can internally infer and retrieve relevant prior results without explicit prompt augmentation. Further, the reliance on large model scales (≥65B parameters) suggests that the emergent benefits of residualized reasoning are contingent on model capacity (Jiang et al., 2023).

Future research may investigate (i) methods to automate identification of optimal residual connection placements, (ii) hybrid symbolic-neural approaches for graph extraction, and (iii) generalization to domains with entangled or cyclic logical graphs.