Rationale-Driven Multi-Task Learning

• Rationale-driven multi-task learning is a framework that incorporates compact, human- or model-generated rationales as auxiliary supervision to improve model interpretability and prediction accuracy.
• It employs architectures like extractor–predictor, teacher–student, and graph-based models to jointly optimize primary predictions and rationale generation.
• Empirical findings demonstrate enhanced accuracy, reduced reliance on extensive annotations, and robust performance across diverse domains such as NLP and clinical decision support.

Rationale-driven multi-task learning encompasses a family of inductive frameworks and optimization strategies that explicitly incorporate rationales—short, human- or model-generated justifications for decisions—into the training pipeline of models addressing complex prediction tasks. Both extractive (token-level) and free-text rationales serve as auxiliary supervision signals, regularized objectives, or conditional generative targets. This paradigm enhances prediction robustness, interpretability, and faithfulness of downstream models, particularly in regimes characterized by limited annotation resources, continual task sequences, or multi-faceted outputs (Bhat et al., 2021, Veerubhotla et al., 2023, Si et al., 2022, Do et al., 28 Feb 2025, Hasan et al., 23 Dec 2025, Xiong et al., 2023, Carton et al., 2021).

1. Core Principles and Definitions

Rationales in machine learning refer to compact, informative excerpts of the input sequence (token spans, sentences, graph subcomponents, or textual explanations) that alone suffice—or at least substantially contribute—to model predictions. Three desiderata widely acknowledged are sufficiency (alone, the rationale yields the correct prediction), completeness (removing the rationale destroys model certainty), and coherence (rationales form contiguous, non-fragmented spans or explanations) (Bhat et al., 2021).

Rationale-driven multi-task learning systems are architected to optimize both primary decision outputs and rationale generation, either in parallel or sequentially. Cross-modal relationships (label-rationale, decision-explanation), sample selection, and task balancing are crucial components. This dual focus induces models to build more causally grounded internal representations, mitigating shortcut learning and enhancing explainability (Xiong et al., 2023, Carton et al., 2021).

2. Model Architectures and Integration Mechanisms

Rationale integration architectures predominantly employ the following paradigms:

• Extractor–Predictor Models: Employ separate rationale extractor heads (e.g., BERT/Gumbel-Softmax for token-level masks) feeding a rationalized input to a predictor. Multi-task coupling is realized via shared encoders and joint or selective loss functions (Carton et al., 2021).
• Teacher–Student Self-Training: Multi-stage flows where a teacher model generates pseudo-labels and pseudo-rationales from limited annotation; a student model trains on these with re-weighted and auxiliary rationale-quality losses. Iterative refinement cycles prevent label drift (Bhat et al., 2021).
• Dual-Teacher Frameworks: Decompose joint decision-rationale generation into two specialist teachers (predictor and rationalizer), whose outputs (pseudo-labels, rationales) are distilled into a multi-task joint student. Conditioning explanations on predicted labels and regularizing faithfulness (via Masked Label Regularization) achieves superior label–rationale alignment (Veerubhotla et al., 2023).
• Seq2Seq Sequential Generation: Models such as RaDME (Rationale-Driven Multi-trait Essay scoring) autoregressively emit a trait score followed by its rationale, ensuring that rationale tokens are conditioned on score tokens through causal decoder attention (Do et al., 28 Feb 2025).
• Graph-Based GCNs with Salience Perturbation: Evidence nodes and adjacency graphs permit rationale extraction as “faithful subgraph” selection, enforced by fidelity, compactness, and topology regularizers alongside claim verification tasks (Si et al., 2022).

This diversity of mechanisms allows flexible adaptation to the requirements of continual relation extraction (Xiong et al., 2023), clinical decision support (Hasan et al., 23 Dec 2025), essay scoring (Do et al., 28 Feb 2025), and multi-hop reasoning (Si et al., 2022).

3. Multi-Task Objectives and Regularization Strategies

The design of multi-task loss functions for rationale-driven training typically encompasses:

Model Family	Primary Task Loss	Rationale Loss / Regularizer	Balancing/Weighting
Teacher–Student	Prediction (cross-entropy)	Token rationale loss, sufficiency, completeness, coherence	Confidence-based re-weighting, λ-schedules
Dual-Teacher/Distill.	Joint prediction–explanation loss	Masked Label Regularization (entropy)	Product of teacher confidences, λ-tradeoff
Extractor–Predictor	Label cross-entropy	Token/sentence rationale loss (weighted, selective supervision)	Class weights, importance embeddings
Graph GCN	Full-graph claim verification	Fidelity, compactness, topology losses	λ_i regularizers (default, ablated)
Seq2Seq	Score classification/MSE	Cross-entropy over rationale sequence	Trait-wise ∝ α_t, β_t
Contrastive Replay	Classification (main, auxiliary)	InfoNCE contrastive loss	λ for contrastive term

Loss balancing is frequently achieved through hyperparameter sweeps, curriculum schedules (e.g., linear α_t increase), and instance-weighted pseudo-labeling. Auxiliary rationale objectives directly encode desiderata such as sufficiency (matching label distributions on rationales), completeness (maximizing uncertainty when rationale is absent), and label–explanation association (via entropy or dependency analyses) (Bhat et al., 2021, Veerubhotla et al., 2023).

4. Training Procedures: Self-Training, Scheduled Sampling, and Replay

Self-training strategies enable label and rationale supervision to scale beyond scarce gold annotation:

• Sample Selection and Confidence Weighting: Instance and token weights derived from model confidences modulate gradient influence from noisy pseudo-labels during student updating cycles (Bhat et al., 2021, Veerubhotla et al., 2023).
• Two-Stage Curricula: Initial rationalization pretraining (Stage-1), followed by joint prediction–rationale training with scheduled sampling, where explanation generation is gradually transitioned from conditioning on gold labels to predicted labels, reducing exposure bias (Hasan et al., 23 Dec 2025).
• Contrastive Replay: Post-task rehearsal on stored exemplar triples ([x, r, y]) using InfoNCE losses and contrastive rationales generated to distinguish analogous classes, thereby mitigating catastrophic forgetting and inter-class confusion (Xiong et al., 2023).
• Graph-Based Regularization: GCN rationale subgraphs learned without annotation, enforced by fidelity, compactness, and topology; optional replacement by gold rationale labels for supervised settings further boosts quality (Si et al., 2022).

Early stopping criteria, especially based on rationale validation F1, are widely recommended to prevent drift and overfitting to pseudo-generated rationales (Bhat et al., 2021).

5. Empirical Findings and Benchmarks

Consistent performance improvements have been reported across domains:

• Teacher–student rationale frameworks yield +6.45% absolute F1 gain over single-task baselines in few-shot NLU, and approach fully supervised performance with an order-of-magnitude fewer labeled examples (Bhat et al., 2021).
• Dual-teacher models with masked label regularization close ~13 percentage-point gaps in accuracy and nearly match supervised BLEU explanation quality; explicit regularization increases label–rationale dependency metrics to ≥88.7% (Veerubhotla et al., 2023).
• RaDME matches or exceeds prior SOTA in multi-trait essay scoring (QWK ≈ 0.711) and demonstrates that rationale supervision improves both accuracy and explanation coherence. Ablations confirm a ≈0.047 decrease without rationale generation (Do et al., 28 Feb 2025).
• Reason2Decide achieves F1 and explanation fidelity gains in both clinical and QA datasets with models 40× smaller than foundation LLMs, confirming LLM-generated rationales suffice for rationale pretraining (Hasan et al., 23 Dec 2025).
• Multi-hop graph models (SaGP) improve rationale extraction/claim verification F1 over previous SOTA, with diagnostic property losses crucial for stability (Si et al., 2022).
• Continual relation learners with multi-task rationale tuning and contrastive replay outperform robust baselines by up to +1.7 points, significant by McNemar tests (Xiong et al., 2023).
• Extractor–predictor models demonstrate that class-weighted rationale supervision and soft importance embeddings are critical for boosting prediction accuracy and rationale recall, especially when human rationales are unevenly informative (Carton et al., 2021).

A plausible implication is the broad transferability of rationale-driven multi-task learning benefits across architectures, modalities, and annotation regimes.

6. Limitations, Robustness, and Practical Guidelines

Key limitations include prompt engineering for rationale generation, computational cost of multi-task and replay routines, and variable quality of human or LLM-provided rationales (Xiong et al., 2023, Carton et al., 2021). Selective supervision or curriculum learning can starve extractors of signal if not carefully managed (Carton et al., 2021). Many systems mitigate exposure bias by introducing scheduled sampling of model predictions during explanation training, and support robustness to rationale sources (human-authored, LLM-generated) (Hasan et al., 23 Dec 2025).

Practical best practices synthesized across works (Bhat et al., 2021, Veerubhotla et al., 2023, Do et al., 28 Feb 2025, Hasan et al., 23 Dec 2025):

1. Jointly predict labels and rationales on limited data using shared encoder architectures for enhanced inductive bias.
2. Leverage self-training and pseudo-labeling with instance weighting for large-scale unlabeled inputs.
3. Explicitly encode rationale desiderata—sufficiency, completeness, and contiguity—via auxiliary losses, rather than depend exclusively on implicit attention mechanisms.
4. Use two-stage or curriculum frameworks to sequentially build explanation and prediction capabilities, mitigating exposure bias.
5. Perform early stopping based on rationale validation F1 as a proxy for model alignment.

This synthesis shows rationale-driven multi-task learning to be a foundational technique for explainable, robust, and accurate machine learning in settings ranging from NLU to medical reasoning, multi-trait scoring, and continual learning.