Self-NLEs: Model-Generated Explanations

• Self-NLEs are model-generated, free-form justifications produced with predictions to reveal internal reasoning and promote transparency.
• They leverage diverse architectures, including joint prediction–explanation models and iterative self-verification, to improve faithfulness and support end-task performance.
• Evaluation combines automatic metrics, attribution tests, and human assessment to balance explanation accuracy, consistency, and real-world applicability.

Self-Natural Language Explanations (self-NLEs) are model-generated, free-form justifications of a model's own outputs, produced as part of or immediately after decision-making. They constitute a primary mechanism by which LLMs and other advanced neural systems seek to provide transparency, interpretability, and user-aligned rationalization for their predictions across a wide array of domains, including NLP, program synthesis, vision-language reasoning, and structured data translation. This article systematically reviews the principles, architectural variants, faithfulness challenges, evaluation strategies, and empirical impact of self-NLEs in current literature.

1. Definitions, Scope, and Motivations

Self-NLEs are distinct from post-hoc explanations and human-annotated rationales: they are produced by the model itself, usually conditioned on its internal representations and decision trajectory, often during inference or as a joint product with its prediction. They may take the form of causal chain-of-thought, feature- or token-level attribution, or high-level summaries referencing model-relevant latent knowledge (Majumder et al., 2021, Wang et al., 28 May 2025, Huang et al., 2023).

Technical motivations for self-NLEs include:

• Transparency: Making opaque decision processes accessible in human-like terms (Majumder et al., 2021, Huang et al., 2023).
• Faithfulness: Ensuring the explanation accurately reflects internal model reasoning, rather than being a plausible but ungrounded post-hoc rationalization (Wang et al., 28 May 2025, Admoni et al., 9 Jun 2025, Bhan et al., 10 Jun 2025).
• Support for end-task performance: Providing actionable, intermediate representations for tasks such as multilingual code translation or complex question answering (Tang et al., 2023, Fan et al., 2024).
• Human-AI collaboration: Facilitating user trust and error detection in decision-intensive domains (Fan et al., 2024).

Variations exist from tightly coupled joint prediction-explanation models (Sammani et al., 2022), to iterative self-refinement frameworks (Wang et al., 28 May 2025), to architectures that explicitly encode and align extractive rationales, knowledge grounding, and free-text generation (Majumder et al., 2021).

2. Principles and Architectural Variants

Self-NLE generation frameworks divide into several main paradigms:

• Joint Prediction and Explanation: Architectures like NLX-GPT (Sammani et al., 2022) and e-UG (Kayser et al., 2021) generate predictions and explanations via a single generative process, often employing shared representations to tightly couple decision and justification.
• Post-hoc Rationalization with In-Loop Feedback: CycleSQL (Fan et al., 2024) generates self-NLEs as feedback, uses them to semantically verify candidate outputs (e.g. SQL queries), and iterates until explanation/decision alignment is achieved.
• Latent Variable Models: ELV and variants (Zhou et al., 2020) model explanations as continuous latent variables jointly with labels, optimizing variational lower bounds via alternating inference and generative steps.
• Self-Critique and Iterative Refinement: SR-NLE (Wang et al., 28 May 2025) employs multi-round, model-internal critique and refinement, using either natural language or feature attribution signals to iteratively improve explanation faithfulness.
• Knowledge-Grounded Rationalization: The RExC framework (Majumder et al., 2021) grounds explanations in both extractive rationales (salient input spans) and selected background-knowledge snippets, training all modules end-to-end for maximal alignment.
• Self-Critical Learning in Vision-Language: S³C (Suo et al., 2023) introduces a reward that scores explanations by their causal consistency with answers, enabling semi-supervised learning without fully annotated explanations.

Representative architectural motifs and their primary characteristics are summarized below.

Model/Framework	Coupling	Explanation Form	Key Mechanism
NLX-GPT, e-UG	Joint	Free-form NLE	Unified encoder–decoder
CycleSQL	Iterative self-verifier	Data-grounded NLE	SQL rewriting + NLI check
RExC	End-to-end multi-branch	NLE + rationale + kn.	Latent selectors, grounding
SR-NLE	Iterative post-hoc	Faithful NLE	Critique & feedback
ELV	Latent variable EM	Latent NLE	Mutual optimization
S³C	Semi-supervised, RL loop	VQA rationales	Self-critical learning

3. Faithfulness and Self-Consistency

A central challenge for self-NLEs is the alignment between the explanation and the model’s true reasoning process. Key findings and technicalities include:

• Faithfulness Criteria: An explanation is faithful if and only if it mentions only those features, evidence, or intermediate steps that the model actually uses to produce its prediction (Bhan et al., 10 Jun 2025, Admoni et al., 9 Jun 2025).
• Quantitative Faithfulness Comparisons: CycleSQL utilizes a feedback verifier based on NLI entailment between self-NLE and the original NL question—only accepting candidate output if semantic entailment is established (Fan et al., 2024). SR-NLE applies counterfactual-based metrics—measuring if the explanation updates when single input tokens change the answer (Wang et al., 28 May 2025).
• Self-Consistency of Explanations: PSCB (Admoni et al., 9 Jun 2025) defined self-consistency via feature-attribution vector similarity between the decision and the explanation. Standard cosine-based metrics tend to collapse the quality range; Spearman rank correlation offers greater sensitivity.
• NeuroFaith Framework: Recent work directly contrasts generated self-NLEs against concepts decoded from intermediate neural activations (e.g., residual stream, MHA blocks), quantifying local and global agreement in multi-hop reasoning tasks (Bhan et al., 10 Jun 2025).
• Empirical Gaps: Even large, instruction-tuned models exhibit significant deviance (NLE-f ≈ 23% vs NLE-c ≈ 50–60%), with "explainer parrots"—where the explanation is correct but unfaithful—comprised 37–40% of cases (Bhan et al., 10 Jun 2025).
• Learning for Faithfulness: Direct Preference Optimization (DPO) using alignment-based preferences (best/worst explanations by Spearman correlation to internal importances) can improve explanation–reasoning consistency without harming task accuracy (Admoni et al., 9 Jun 2025).

4. Evaluation Strategies and Benchmarks

Self-NLEs are subject to diverse evaluation protocols:

• Automatic Metrics: Comprise both general NLG scores (BLEU, ROUGE, CIDEr, BERTScore) and explanation-specific measures, such as:
  • Feature/comprehensiveness: accuracy drops from masking out most-important tokens (Majumder et al., 2021, Huang et al., 2023).
  • Sufficiency: confidence gap when only most-important tokens are kept (Huang et al., 2023).
  • Consistency: attribution overlap (cosine, Spearman) between decision and explanation (Admoni et al., 9 Jun 2025).
  • Counterfactual Tests: faithfulness under local, behavioral input perturbations (Wang et al., 28 May 2025).
• Human Evaluation: Quality, trustworthiness, and interpretability are rated in both task-specific settings (e.g., explaining VQA answers in e-ViL (Kayser et al., 2021), SQL query outputs in CycleSQL (Fan et al., 2024)) and generic scoring (coherence, informativeness).
• Self-Evaluation Frameworks: Some approaches (e.g., “explain-predict” in NLX-GPT (Sammani et al., 2022)) train auxiliary classifiers to check whether the explanation alone allows prediction recovery, while "retrieval-based attacks" probe for data/model bias in explanation generalization.
• Benchmarks/Datasets: e-ViL (Kayser et al., 2021) and e-SNLI-VE (Kayser et al., 2021) for vision-language, PSCB (Admoni et al., 9 Jun 2025) for post-hoc LLM QA explanations, MultiPL-C2C (Tang et al., 2023) for code translation, eSNLI (Majumder et al., 2021, Wang et al., 28 May 2025, Wang et al., 1 Jan 2026) for NLI NLEs, etc.

5. Domain-Specific Instantiations and Use Cases

Self-NLEs have been adapted to a range of modalities and tasks:

• Vision-Language Reasoning: Unified models (e.g., NLX-GPT, e-UG) couple visual reasoning, answer prediction, and explanation generation, achieving both state-of-the-art performance and high human-judged explanation quality (Sammani et al., 2022, Kayser et al., 2021).
• Program Synthesis and Translation: "Explain-then-Translate" introduces a model-generated explanation as an intermediate representation for code-to-code translation, yielding up to +12% pass@1 gains, especially for difficult or low-resource language pairs (Tang et al., 2023).
• Structured Data Querying: CycleSQL introduces data-grounded self-NLEs for NL-to-SQL translation, utilizing provenance tracking and semantic rewriting to validate and improve output selection via an entailment feedback loop (Fan et al., 2024).
• Feature Attribution in Sentiment Analysis: ChatGPT and similar instruction-tuned LLMs can output token-level importances as self-NLEs, with faithfulness parity compared to occlusion or LIME but differing substantially in agreement structure and granularity (Huang et al., 2023).
• Vision QA and Explanatory Multi-modal Tasks: S³C (Suo et al., 2023) combines self-critical reward and semi-supervised learning to align free-text rationales with answer predictors, outperforming specialized models under both human and automatic metrics.

6. Robustness, Limitations, and Practical Guidance

Empirical investigations uncover both the robustness and shortcomings of current self-NLE frameworks:

• Model Compression Effects: Quantization slightly but consistently reduces self-NLE plausibility and faithfulness (up to 4.4% and 2.38%, respectively). Counterfactual explanations tend to be more robust than free-form NLEs. No quantization technique universally dominates (Wang et al., 1 Jan 2026).
• Sample Efficiency and Training Regimes: Models such as ELV (Zhou et al., 2020) and SR-NLE (Wang et al., 28 May 2025) demonstrate that high faithfulness can be achieved with relatively few human explanations and, in some cases, purely self-generated feedback.
• Trade-offs: Performance–explanation synergy is not always realized—certain settings show explanation generation slightly degrades base task accuracy (e.g., -4% in sentiment classification (Huang et al., 2023)), and faithfulness metrics may not correlate with correctness (Admoni et al., 9 Jun 2025).
• Evaluation Challenges: Existing metrics disagree, and evaluation frameworks relying on input perturbation or attribution are sometimes insensitive to explanation granularity or ill-adapted to LLMs’ behaviors (Huang et al., 2023, Bhan et al., 10 Jun 2025).

Practical Recommendations (synthesized from multiple studies (Wang et al., 1 Jan 2026, Wang et al., 28 May 2025, Fan et al., 2024)):

• Empirically validate explanation quality and faithfulness for each application and model variant.
• Consider integrating counterfactual or feature attribution–based self-critique in iterative refinement.
• Prefer explanation forms most robust to perturbations (e.g., counterfactuals for compressed models).
• Combine automatic, attributional, and human evaluations for deployment in safety-critical domains.

7. Conclusions and Future Directions

Recent advances in self-NLEs have produced frameworks that jointly optimize for performance, faithfulness, and user-aligned interpretability across natural language, vision-language, structured data, and programming tasks. The leading edge is moving towards tighter coupling of explanations and internal model evidence, via direct neural interpretation (Bhan et al., 10 Jun 2025), large-scale self-consistency training (Admoni et al., 9 Jun 2025), and data-provenance-grounded justification (Fan et al., 2024).

Future challenges include:

• Faithfulness–Performance Coupling: Closing the still significant gap between explanation correctness and true internal faithfulness, especially in large foundation models (Bhan et al., 10 Jun 2025).
• Multi-modal and Causal Faithfulness: Extending neuro-symbolic, chain-of-thought, and concept-level metrics to multi-modal or reasoning-intensive settings (Sammani et al., 2022, Suo et al., 2023).
• Online/Continuous Faithfulness Training: Incorporating dynamic consistency and faithfulness signals into streaming or in-the-loop model updates (Admoni et al., 9 Jun 2025).
• Human-AI Coordination: Addressing explanation utility and trust calibration in human-in-the-loop settings, especially where imperfect explanations may mislead users (Fan et al., 2024, Huang et al., 2023).

The self-NLE paradigm is central to the interpretability agenda for LLMs and complex neural models, with ongoing research targeting greater reliability, measurable faithfulness, and seamless integration with end-task utility.