LLM-Graded Interpretability Tests

• LLM-Graded Interpretability Tests are automated, model-driven techniques that assess an artifact’s clarity by using LLMs to query, reconstruct, and evaluate reasoning processes.
• They follow a structured workflow—rendering artifacts, querying with an LLM, and applying automated grading metrics such as robustness and phrase sensitivity.
• Adopted across domains like mathematics and program policy, these tests enhance scalability and objectivity by replacing subjective human evaluations with precise, metric-based analysis.

LLM-graded interpretability tests are a suite of automated, model-driven techniques designed to evaluate how well an LLM can explain, simulate, or reconstruct target artifacts—such as mathematical solutions, programmatic policies, and learned models—using only its own reasoning processes, without human intervention. These tests quantify interpretability not as a property of artifact structure alone, but as the empirical ability of an LLM to process representations and answer probing questions or perform reconstruction tasks with measurable fidelity. The methodology has been adopted and extended in diverse domains, including mathematics education, program policy analysis, and agentic machine learning tool design (Dakshit et al., 19 Oct 2025, Bashir et al., 2023, Singh et al., 5 May 2026).

1. Core Principles and Motivation

LLM-graded interpretability tests are motivated by the limitations of traditional metrics that emphasize correctness or performance at the expense of reasoning process, conceptual clarity, or transparency. Unlike human-centric interpretability studies—which rely on user surveys, expert labeling, or subjective quality ratings—LLM-graded approaches automate the evaluation of interpretability by treating the LLM itself as both the explainer and evaluator. The fundamental assumption is that if an advanced LLM can reliably answer structured queries about a solution, model, or program based solely on its representation, the underlying artifact possesses a form of operational interpretability that is both scalable and testable (Bashir et al., 2023, Singh et al., 5 May 2026).

2. Methodological Frameworks

Across domains, the standard LLM-graded interpretability workflow comprises three phases:

1. Rendering: The artifact (e.g., LLM solution, programmatic policy, fitted model) is rendered into a string-based format (plain-text chain-of-thought, DSL code, or model summary).
2. LLM Querying: An LLM receives the rendered representation, along with a template question or reconstructive task, and generates a response.
3. Automated Grading: The output is parsed and compared to a reference or computed ground truth, with pass/fail or scalar metrics defined by strict criteria.

Exemplars by Domain

• Mathematical Reasoning: Decomposing multistep solutions into semantically labeled steps, annotating operations and concepts, constructing reasoning-flow graphs, and probing solution robustness via prompt ablation (Dakshit et al., 19 Oct 2025).
• Programmatic Policy Interpretability: The LINT methodology, where one LLM explains source code in natural language under constraints, a second LLM reconstructs code from that explanation, and the behavioral equivalence of original and reconstructed code is measured on structured benchmarks (Bashir et al., 2023).
• Simulatable Model Tests: For agentic-imodels, interpretability is defined as the LLM's ability to answer automatically gradeable prompts about a model's behavior, predictions, and structure using only its string representation (Singh et al., 5 May 2026).

3. Metric Definitions and Formalization

A broad class of LLM-graded interpretability metrics emerges from these frameworks, typically blending discrete task success rates with continuous similarity measures.

Structured Metrics in Key Frameworks

Domain	Metric/Score	Formal Definition
Math Reasoning	Reasoning Complexity	$\overline{C} = \frac{1}{N}\sum_{i=1}^N c_i$, with $c_i$ indicating complexity of step $i$
	Phrase Sensitivity	$$\mathrm{PS} = 1 - \frac{|\{p:\mathrm{sim}(\mathrm{orig}, \mathrm{abl}_p)\ge\tau\}|}{\{\text{all phrases}\}}$$
	Robustness	$R = \frac{1}{M} \sum_{i=1}^M I_i$, $I_i = 1$ if $\mathrm{sim}(O_0, O_i)\ge\tau$
Programmatic Policy	LINT	$$I(\pi) = B(\pi, R_{\mathrm{LLM}_2}(E_{\mathrm{LLM}_1}(\pi,G,C),G))$$
Model Simulatability	$S_{\mathrm{sim}}$	$$S_{\mathrm{sim}} = \frac{1}{N}\sum_{i=1}^{N} \mathbb{1}\{a_i = \text{correct}_i\}$$

In all cases, interpretability is treated as an empirical property bound to a question-and-answer or reconstruction protocol mediated by the LLM, with outputs compared to ground truth via edit distance, cosine similarity, pass/fail thresholds, or behavioral equivalence.

4. Implementation Protocols and Test Suites

Mathematical Solutions

Interpretability tests for LLM-generated multistep solutions (Dakshit et al., 19 Oct 2025):

• Reasoning-Flow Extraction: Rule-based segmentation of solution steps, semantic annotation (operation type, conceptual tag, cognitive complexity), graph construction of reasoning dependencies, and alignment to reference solutions.
• Prompt Ablation: Atomic elements (numbers, keywords, symbols) are systematically masked; the LLM is prompted on each ablated variant; semantic and syntactic divergences in the outputs are measured and aggregated into phrase sensitivity and robustness scores.

Program Policies and Models

• LINT Pipeline (Bashir et al., 2023): Source code and DSL description are provided to an "explainer" LLM, which outputs a jargon-free, high-level explanation. This explanation is provided to a "reconstructor" LLM, which attempts to generate code functionally equivalent to the original. Behavioral similarity is computed via structured test suites (e.g., input-output agreement, action trace similarity). Multiple reconstructions mitigate random success.
• Agentic-imodels Simulatability (Singh et al., 5 May 2026): Test harnesses generate synthetic data, fit models, produce string representations, and pose standardized questions in six categories (feature attribution, point simulation, sensitivity analysis, counterfactual reasoning, structural understanding, nonlinear function simulation). Grading is fully automated, requiring only the LLM's answer and a reference solution.

5. Experimental Findings and Applications

Empirical results demonstrate the sensitivity and discriminative power of LLM-graded interpretability tests:

• Calculus Solutions: Human accuracy ~89% vs. baseline LLM ~63% on Calculus I; robustness and phrase sensitivity distinguish between RAG, contextual retrieval, and prompt variants; ablation highlights model brittleness, especially to specific keywords (Dakshit et al., 19 Oct 2025).
• Policy Reconstruction: Perfect LINT scores for simple, non-obfuscated code; near-zero for heavily obfuscated IOCCC programs; strong monotonicity in behavioral metrics with increased code obfuscation. LINT outperforms action and feature similarity baselines in MicroRTS policy reconstruction (Bashir et al., 2023).
• Simulatable Model Benchmarks: Evolved regressors show high $c_i$0 while retaining competitive RMSE ranks; best models surpass baselines by coupling compact representations with structural regularization; correlation between development and hold-out interpretability scores ($c_i$1) indicates robustness against test-overfitting (Singh et al., 5 May 2026).

6. Strengths, Limitations, and Domain Generalization

Strengths

• Fully automated: eliminates the need for human raters, enabling scalable benchmarking.
• Proxy of agent-readability: reflects whether an artifact can be "consumed" and acted on by advanced AI systems, supporting both introspective and agentic pipelines.
• Modular and extensible: can be adapted to arbitrary DSLs, model classes, or reasoning domains by altering question templates and grading logic.

Limitations

• Dependent on the LLM’s domain competence; fails if the artifact or DSL is not within LLM prior knowledge.
• Subject to prompt design; constraints must prevent code leakage or tautological answers to preserve fidelity.
• Proxy for, not substitute of, end-user interpretability: LLM success does not guarantee clarity for human novices unless alignment is validated.

Extensions

Potential extensions include human-in-the-loop verification, variable-strength abstraction constraints, chain-of-thought elicitation in explanations, and adaptation to domains such as reinforcement learning, visual reasoning, or scientific computation (Bashir et al., 2023, Singh et al., 5 May 2026).

7. Pedagogical and Agentic Implications

LLM-graded interpretability tests have immediate applications in education and explainable agency:

• Formative Feedback: Students and instructors can visualize LLM reasoning graphs or ablation maps to diagnose conceptual errors and identify learning "hooks" (Dakshit et al., 19 Oct 2025).
• Model Evolution: Agentic autoresearch loops employ LLM-graded metrics directly as objectives, iteratively evolving models that optimize both predictive power and simulatability under LLM scrutiny (Singh et al., 5 May 2026).
• Policy Safety and Verification: Ability to automatically diagnose and reconstruct interpretable policies from agent-readable documentation or code enhances transparency and verifiability in AI deployment (Bashir et al., 2023).

A plausible implication is that as LLM-graded interpretability becomes increasingly robust, automated agents will move from using solely human-interpretable tools to iteratively designing artifacts whose structure, rationale, and predictive logic are optimized for both human and agentic readability.