RationaleTS: Minimal Rationales for ML

• RationaleTS is a method that defines minimal, sufficient token subsets as rationales to explain and optimize model predictions in sequential tasks.
• It leverages a greedy NP-hard algorithm and word-dropout fine-tuning to align generated rationales closely with human-like justifications.
• Validated across domains like NLU, text-to-SQL, and time series reasoning, RationaleTS boosts task accuracy and transparency.

RationaleTS is a category-defining family of methods leveraging model-generated or dataset-derived rationales—minimal, sufficient subsets of context or stepwise reasoning chains—to optimize interpretability and faithfulness in sequential prediction, NLU, program synthesis, multi-trait scoring, and time series reasoning tasks. RationaleTS frameworks combine principled combinatorial objectives, efficient greedy or retrieval-based algorithms, and, where necessary, architectural or fine-tuning modifications to enable rationale-based explanations that align closely with human justifications and enhance downstream task accuracy.

1. Formalization and Theoretical Objective

The foundational RationaleTS formulation, from the context of sequential prediction, defines the rationale for a token $y_t$ as the smallest set $S \subseteq \{1,\dots,t-1\}$ of prefix tokens such that the model $f$'s top prediction on $y_S$ matches that of the full prefix $y_{<t}$. Formally, the rationale $S^*$ is the solution to:

$$S^* = \arg\min_{S \subseteq [t-1]} |S| \ \text{s.t.} \ \arg\max_{y'} f(y'|y_S) = \arg\max_{y'} f(y'|y_{<t}) = y_t.$$

Alternatively, this can be written as seeking $S$ such that:

$$\min_{S\subseteq[t-1]}\;|S|\quad\text{s.t.}\quad f(y_t\mid y_S)\ge f(y'\mid y_S)\;\forall y'.$$

This minimum sufficient rationale concept generates explanations tightly coupled to model decision-making process, rather than relying on post-hoc heuristics (Vafa et al., 2021).

2. Algorithmic Approaches: Greedy Rationalization and Approximate Optimization

Since the above minimization is NP-hard (set cover reduction), RationaleTS adopts a greedy algorithm:

1. Initialize $S^{(0)}=\emptyset$ (or $\{t-1\}$ for efficiency).
2. Until $y_t$ becomes the model’s top prediction given $y_{S^{(n)}}$,
   • At each iteration, choose $$k^* = \arg\max_{k\in [t-1]\setminus S^{(n)}} f(y_t | y_{S^{(n)} \cup \{k\}})$$ and update $S^{(n+1)} = S^{(n)} \cup \{k^*\}$.
3. Output $S$ as the rationale.

Each step selects the context token with the highest marginal contribution to the conditional probability of $y_t$. This procedure yields an $\ln t$ approximation in typical regimes and exhibits acceptable $O(t^3)$ runtime for transformers at practical context sizes (Vafa et al., 2021).

Related settings, such as chain-of-thought rationalization for SQL synthesis, employ dynamic few-shot prompting and rationalization model fine-tuning to induce multi-step, human-interpretable rationales (e.g., intermediate SQL fragments and English justifications) in a scalable way (Rossiello et al., 10 Feb 2025).

3. Model Compatibility and Fine-Tuning for Faithful Rationalization

A critical requirement for RationaleTS is that the underlying model produces compatible conditional distributions on arbitrary partial contexts. This is formalized as:

$$f(y_t\mid y_S) \approx \sum_{\tilde y_{[t-1]\setminus S}} f(y_t\mid y_{<t})\,\prod_{i\notin S}f(y_i\mid y_{<i}) = p(y_t\mid y_S).$$

Standard MLE training enforces compatibility only for complete contexts. RationaleTS mitigates incompatibility via a "word-dropout" fine-tuning objective, defined as:

$$\mathcal L_{\mathrm{drop}} = -\mathbb{E}_{y_{1:T} \sim F} \mathbb{E}_{S \sim \mathrm{Unif}(2^{[T]})} \sum_{t=1}^T \log f(y_t \mid y_{S_{<t}}).$$

Mixing this with the base MLE, the model is exposed to a spectrum of partial prefixes, which calibrates predictions for incomplete inputs and enables valid, faithful greedy rationales across diverse tasks and architectures (Vafa et al., 2021).

4. Practical Domains and Instantiations

RationaleTS has been instantiated across several domains:

• Sequential prediction (LM / MT): Greedy rationalization for LMs and MT models on OpenWebText, Lambada, and IWSLT’14 (Vafa et al., 2021).
• Few-shot NLU: Joint multi-task teacher-student architectures with pseudo-labeled rationales and auxiliary losses for rationale sufficiency, completeness, and coherence, enabling state-of-the-art few-shot NLU performance (Bhat et al., 2021).
• Program synthesis (text-to-SQL): Multi-stage rationalization with chain-of-thought annotations, dynamic teacher-guided few-shot distillation, and rationalization model fine-tuning, achieving superior execution accuracy especially for complex queries on BIRD (Rossiello et al., 10 Feb 2025).
• Multi-trait essay scoring: Trait-wise rationale generation via LLM prompting, followed by S-LLM fine-tuning on concatenated essay + rationale inputs for trait-interpretable scoring boosts on ASAP/Feedback Prize (Chu et al., 2024).
• Time series reasoning: In multimodal MLLMs, RationaleTS introduces label-conditioned, evidence-to-implication rationale generation, hybrid retrieval (temporal and semantic), and retrieval-augmented in-context prediction, delivering robust gains across finance, traffic, and power datasets (Liu et al., 6 Jan 2026).

5. Experimental Evidence and Comparative Analysis

Evaluation approaches span both faithfulness metrics and task accuracy:

• Rationale fidelity: Approximation of minimum rationale size ($|S_{\text{method}}|/|S^*|$), with greedy rationalization achieving $\approx1.2\times$ the optimal, sharply outperforming gradient- and attention-based baselines (often $>2\times$) (Vafa et al., 2021).
• Faithfulness: Greedy rationales avoid distractor tokens ($\simeq43\%$ IOU vs. $<15\%$ for attention/gradients in synthetic tests) and achieve top alignment accuracy in MT (Vafa et al., 2021).
• Human plausibility: On annotated Lambada and MT alignments, greedy rationales maximize IOU, F1, and AER while maintaining brevity (Vafa et al., 2021).
• Downstream accuracy: Rationalization models in text-to-SQL boost execution accuracy on challenging splits, with CoT-augmented models outperforming non-CoT baselines by up to $+4.86$ points on hard queries (Rossiello et al., 10 Feb 2025).
• Trait scoring (QWK): LLM rationale-augmented multi-trait scorers increase QWK by $+1.9\%$ to $+16\%$ across S-LLMs and tasks, confirming the signal quality of LLM rationales (Chu et al., 2024).
• Time series F1/AUC: RationaleTS improves F1/AUC by $3$–$5$ points over ICL/LLM baselines on multiple domains—finance, traffic, power—demonstrating the generality of rationale-grounded in-context learning (Liu et al., 6 Jan 2026).

6. Methodological Innovations: Retrieval, Fusion, and Losses

RationaleTS adapts to each domain with methodologically tailored constructs:

• Hybrid retrieval (time series): Data-centric (TabPFN embedding) and semantic (text embedding) similarity are fused (weighted sum; $\lambda=0.8$ empirically optimal) to select informative, label-conditioned rationales (Liu et al., 6 Jan 2026).
• Multi-component losses (few-shot NLU): Incorporation of sufficiency, completeness, and coherence losses focuses student models on concise, necessary, and contiguous rationales (Bhat et al., 2021).
• Dynamic few-shot distillation (text-to-SQL): Iterative, similarity-based selection and execution-validated augmentation progressively expand rationale coverage with high-quality CoT exemplars (Rossiello et al., 10 Feb 2025).
• Cross-modal fusion layers (multi-trait scoring): Rationale and essay representations are integrated with simple or projected fusion mechanisms, with potential for further architectural enrichment (Chu et al., 2024).

7. Limitations, Interpretability, and Open Directions

RationaleTS elucidates model decisions and increases transparency, but several open research axes remain:

• Modality and domain transfer: Empirical coverage is strongest in English-language, structured-text, and time series data; less is known for ablation in other domains (Chu et al., 2024).
• Efficiency: Rationale generation or retrieval—especially at large scale or for long documents—remains a computational bottleneck, though precomputing embeddings or rationale banks yields significant savings (Liu et al., 6 Jan 2026).
• Human alignment: Plausibility studies show alignment with human rationales, but formal user studies quantifying interpretability and trust remain uncommon.
• Fusion architecture: Deeper integration of rationale-generation and prediction (e.g., via multitask or gated fusion networks) is suggested as a promising avenue (Chu et al., 2024).
• Sensitivity: Selection of $K$ (number of retrieved rationales) and $\lambda$ (retrieval fusion weight) impacts performance, with under/over-retrieval and improper weighting causing drop-offs in accuracy (Liu et al., 6 Jan 2026).
• Potential copying: In context conditioning, including true labels in prompts can artificially inflate performance as models may copy labels rather than reason (Liu et al., 6 Jan 2026).

RationaleTS thus provides a suite of principled, empirically validated algorithms for rationale discovery and utilization, bridging classical explanation desiderata with the operational needs of contemporary neural models across sequence, structure, trait, and time series domains (Vafa et al., 2021, Bhat et al., 2021, Rossiello et al., 10 Feb 2025, Chu et al., 2024, Liu et al., 6 Jan 2026).