Causal Stepwise Evaluation (CaSE)

• Causal Stepwise Evaluation (CaSE) is a method that assesses intermediate reasoning steps in LLMs by evaluating each step with only its preceding context.
• It decomposes reasoning quality into relevance and coherence, preventing hindsight bias and enabling targeted error diagnosis.
• CaSE is practically applied for debugging, training data curation, and model improvements, demonstrating measurable accuracy gains in LLMs.

Causal Stepwise Evaluation (CaSE) is a principled evaluation methodology for analyzing the quality of intermediate reasoning steps in LLMs and other computational reasoning systems. Unlike final-answer evaluation paradigms that provide a binary or aggregate signal, CaSE decomposes the reasoning trace into discrete steps and evaluates the quality of each step in a manner that is causally faithful to the model’s generative process. This is achieved by constraining the evaluation of step $k$ to only the question and the stepwise context up to step $k-1$, mimicking the auto-regressive, history-only access typical in LLMs. CaSE thus removes hindsight bias, enables diagnosis of intermediate failures in reasoning chains, and offers a rigorous foundation for curating training data and driving improvements in reasoning-capable systems (Do et al., 23 Oct 2025).

1. Causal and Stepwise Evaluation Paradigm

CaSE shifts the evaluation focus from end-to-end correctness to an auto-regressive, context-limited analysis of reasoning traces. Given a question $Q$ and a reasoning trace $[{\rm Step}_1, {\rm Step}_2, \ldots, {\rm Step}_n]$, CaSE assesses each ${\rm Step}_k$ only with respect to its context $$\mathcal{C}_{<k} = \{{\rm Step}_1, ..., {\rm Step}_{k-1}\}$$ and $Q$, using explicitly specified reasoning aspects.

Formally, for any aspect $a$ (such as relevance or coherence), CaSE defines the evaluation operator as: $$\text{Eval}_{\text{aspect}}({\rm Step}_k \mid Q, \mathcal{C}_{<k})$$ This formulation prevents any future information leakage—each step is judged only on what would be available to a model at generation time up to that point.

The significance of this paradigm is multi-fold:

• It matches the generation process of LLMs (which have no access to unseen, future steps).
• It avoids inflated coherence or informativeness judgment that arises in full-trace, hindsight-biased scoring.
• It enables step-level debugging and targeted improvements at precise points in the reasoning chain.

2. Reasoning Quality Dimensions: Relevance and Coherence

CaSE decomposes reasoning quality into at least two axes:

• Relevance: Measures whether a step addresses the core question and meaningfully contributes to problem resolution. An irrelevant step introduces noise, redundancy, or spurious associations.
• Coherence: Measures whether a step logically follows from its immediate predecessors. Logical breaks, unjustified leaps, or contradictions degrade coherence.

Expert annotators or LLM-based evaluators apply these criteria at each step. Critically, for solution-level scoring, the method takes a conjunctive approach: a solution is labeled as relevant (or coherent) only if all steps meet the standard for that aspect.

Empirical analysis (see (Do et al., 23 Oct 2025), Figures 1–2) shows that solutions with consistently high stepwise relevance and coherence—even if the final answer is wrong—tend to be on the correct problem-solving trajectory, indicating these metrics’ strong diagnostic power.

3. Implementation Methodology

CaSE operationalizes this evaluation via a three-component pipeline:

a. Causally Constrained Assessment: For each $k$, the evaluator is presented $(Q, \mathcal{C}_{<k}, {\rm Step}_k)$ and asked to rate ${\rm Step}_k$ for specified aspects (typically as a binary label but extensible to graded or multi-dimensional assessment).

b. Aspect-Specific Evaluation via LLM Prompts: Evaluation can be performed by humans or by LLM-based evaluators with prompting designed to ensure strict context access—never leaked or post hoc information (see (Do et al., 23 Oct 2025), Figure 1).

c. Multi-Aspect Aggregation: Final solution quality is measured with conjunctions or logical aggregation over individual step scores. The method also supports aspect-specific partial credit or error chain localization.

This process is implemented in expert-annotated benchmarks such as MRa-GSM8K (math word problems) and MRa-MATH, where step segmentation guidelines are strictly followed.

4. Empirical Validation and Benchmarks

CaSE’s validation is twofold:

• Agreement with Human Judgments: On MRa-GSM8K and MRa-MATH, CaSE achieves higher alignment with expert ratings compared to full-trace or best-of-N evaluation strategies.
• Predictive Power: Chains rated highly in relevance and coherence at the step level are more likely to contain correct final answers, demonstrating that these intermediate signals carry meaningful information about reasoning trajectory and reliability.

Application to contemporary LLMs of various sizes demonstrates the discriminative power of CaSE: models that score well on CaSE-judged relevance and coherence perform better on downstream tasks.

5. Impact on Model Training and Performance

CaSE has demonstrated utility both for inference-time intervention and for training data curation:

• Aspect-Guided Prompting: Dedicated prompts emphasizing relevance/coherence directly improved LLM final answer accuracy (by ~1.1 points on average on several tasks).
• Supervised Fine-Tuning Data Curation: Filtering SFT training data by step-level CaSE assessments (at step or sample level) produces datasets (e.g., CaSE-1K) that yield LLMs with markedly improved accuracy, relevance, and coherence on benchmarks such as AIME24 and GSM8K.

This establishes that stepwise causal evaluation is not merely diagnostic but also practically effective for dataset quality control and downstream performance improvement.

6. Practical Applications and Debugging Utility

CaSE exposes granular patterns of reasoning failure not accessible with final-answer or aggregate correctness metrics:

• Stepwise scores reveal where reasoning chains deviate, allowing model builders to identify logical gaps, redundancy, or off-topic detours.
• Diagnosis of specific errors (e.g., the first incoherent step that leads to an incorrect answer) supports directed remediation and model editing.
• Step-level analytics can guide targeted model improvements, e.g., by augmenting data at high-failure points or by adjusting architecture/decoding procedures.

For real-time applications, CaSE’s design could be integrated into probabilistic or interactive inference pipelines to regulate reasoning in situ.

7. Extensions, Limitations, and Future Directions

The CaSE framework, as published, focuses on binary axes of relevance and coherence. Future directions outlined in (Do et al., 23 Oct 2025) include:

• Refinement and expansion of evaluation dimensions—adding efficiency, robustness, or process adherence.
• Generalization across additional domains beyond mathematics, such as open-domain QA or multimodal reasoning.
• Integration into live feedback loops, enabling procedural guidance during inference, not just during training or retrospective analysis.
• Direct comparison and possible merging with alternative evaluation paradigms (e.g., process-level error detection, efficiency-oriented scoring) for comprehensive assessment of what constitutes "good" reasoning.

A possible limitation is that step segmentation and granular annotation may require careful human curation or highly controlled LLM-based evaluators to maintain reliability and avoid ambiguous ratings, especially in long or complex traces.

---

In summary, Causal Stepwise Evaluation (CaSE) offers a rigorously defined, causally faithful, and stepwise evaluation protocol for analyzing reasoning traces in LLMs. By grounding evaluation at each point in the generative context and focusing on the axes of relevance and coherence, it enables fine-grained diagnosis, more effective model and data improvement, and a nuanced understanding of what constitutes robust reasoning in state-of-the-art AI systems. This framework provides a scalable and theoretically justified methodology for the advancement, analysis, and training of reasoning-centered models (Do et al., 23 Oct 2025).