VTune: Consistency-Aware Instruction Tuning

• VTune standardizes instruction tuning by unifying format and style, thereby enhancing LLM generalization and robustness.
• It employs automatic format transfer via in-context learning and perplexity-based candidate selection to efficiently harmonize diverse data sources.
• SCAR leverages stylistic and creativity consistency metrics to select high-quality data subsets that match or outperform full-data baselines.

Consistency-Aware Instruction Tuning (VTune) is an approach that emphasizes the critical importance of stylistic, structural, and format uniformity in the instruction-tuning process for LLMs. It addresses both data-level heterogeneity—especially format and presentation inconsistencies across sources—and emergent stylistic factors such as linguistic form and model-internal “instructional surprisal.” Recent research demonstrates that format and style consistency are crucial for generalization, data efficiency, and robust downstream behavior in SFT (supervised fine-tuning) settings (Liang et al., 2023, Li et al., 2024).

1. Formalization of Consistency in Instruction Tuning

Instruction tuning is the process by which LLMs are trained on datasets of the form $$x = (\mathit{inst},\,\mathit{input},\,\mathit{output})$$, where $\mathit{inst}$ represents a rendered instruction in a template format $\mathcal{F}$. In multi-source setups, distinct instruction-tuning corpora employ diverse formats: $\mathcal{F}_s \neq \mathcal{F}_t$ (e.g., task-level, instance-level, keyword-level paradigms). Format inconsistency occurs when the model is trained on format $\mathcal{F}_s$ but must parse $\mathcal{F}_t$ at test time. A canonical metric to quantify format-induced OOD (out-of-distribution) effects is the standard language-model perplexity: $$\mathrm{PPL}(\mathit{inst},\mathit{input}) = \exp\left(-\frac{1}{L} \sum_{i=1}^L \log p_\theta(y_i | \mathit{inst}, \mathit{input}, y_{<i})\right)$$ A significant PPL increase on $\mathcal{F}_t$-formatted examples signals format-induced generalization degradation (Liang et al., 2023).

Separately, “style consistency” encompasses presentation style—surface-level linguistic form, formatting, lexical diversity—and “creativity” or instructional surprisal, formally encoded via features $\mathbf{v}_p$ (presentation embedding via stylometric measurement and encoder pooling) and $\mathbf{v}_c$ (creativity embedding from conditional negative log-likelihood and pooled CLS representations) (Li et al., 2024).

2. Unified Instruction Tuning (UIT): Framework and Workflow

The UIT framework comprises two principal stages: automatic format transfer to enforce homogeneity, and instruction tuning on unified data (Liang et al., 2023).

2.1 Automatic Format Transfer via LLM In-Context Learning

• Seed Set Construction: Select $\mathit{inst}$0 parallel examples $\mathit{inst}$1 mapping source $\mathit{inst}$2 to target $\mathit{inst}$3.
• Format Conversion: For a new $\mathit{inst}$4, prompt GPT-3.5 (API) with the seed set to generate $\mathit{inst}$5 in $\mathit{inst}$6.
• Candidate Denoising: For each source instance, sample $\mathit{inst}$7 candidates $\mathit{inst}$8, then select $\mathit{inst}$9, where perplexity is computed using a local GPT-J checkpoint.

2.2 Pseudocode Illustration

$\mathcal{F}_s$9 (Liang et al., 2023)

3. Data Selection via Style Consistency: SCAR

SCAR (Style Consistency-Aware Response Ranking) extends consistency awareness from format templates to intrinsic stylistic response properties. It operationalizes two orthogonal axes:

• Presentation Consistency: Dense max-pooled representations $\mathcal{F}$0 of stylometric features, with pairwise Euclidean distances $\mathcal{F}$1 denoting stylistic similarity.
• Creativity Consistency: Instructional surprisal $\mathcal{F}$2, with pooled embeddings $\mathcal{F}$3.

A neural reward function $\mathcal{F}$4 scores and ranks candidate pairs. Data selection proceeds by quality-threshold-filtering (helpfulness/correctness) and ranking by $\mathcal{F}$5. The pairwise ranking loss enforces

$\mathcal{F}$6

with triplet style-disentanglement constraints. After training, subsets with top $\mathcal{F}$7 scores are selected for SFT, typically matching or surpassing full-data baselines with only $\mathcal{F}$8 of the data (Li et al., 2024).

4. Empirical Evaluation: Quantitative Gains

4.1 UIT and Format Consistency

UIT improves OOD EM (Exact Match) by $\mathcal{F}$93.6 pp over heuristics and $\mathcal{F}_s \neq \mathcal{F}_t$033 pp over raw mismatched formats in zero-shot settings on T5-LM-xl. Aggregated across different format families (DP, DPE, DPN, DPNE), UIT yields averaged improvements of $\mathcal{F}_s \neq \mathcal{F}_t$1 pp in EM and $\mathcal{F}_s \neq \mathcal{F}_t$2 pp in ROUGE-L over mixed-format baselines. Training-time unification confers a consistent $\mathcal{F}_s \neq \mathcal{F}_t$3 pp EM gain even when merging diverse sources. An offline GPT-J model, distilled from GPT-3.5 format-transfer data, narrows the gap to $\mathcal{F}_s \neq \mathcal{F}_t$40.3 pp EM, substantially reducing inference costs (Liang et al., 2023).

4.2 SCAR and Style Consistency

SCAR-selected subsets (top $\mathcal{F}_s \neq \mathcal{F}_t$5 by style-consistent reward) enable fine-tuned LLMs to match or outperform models trained on the full set in coding (HumanEval/MultiPL-E) and open QA (AlpacaEval). For instance, with only $\mathcal{F}_s \neq \mathcal{F}_t$6 of data, SCAR achieves a win-rate of $\mathcal{F}_s \neq \mathcal{F}_t$7 versus $\mathcal{F}_s \neq \mathcal{F}_t$8 (best non-SCAR baseline) on open QA; for coding, $\mathcal{F}_s \neq \mathcal{F}_t$9k examples selected by SCAR match or exceed the $\mathcal{F}_s$0k-example baseline. Ablations confirm the necessity of quality filtering, triplet disentanglement, and the presence of “referenced” synthetic examples for consistent gains (Li et al., 2024).

5. Analyses: Target-Format, Scaling, and Ablations

• Target Format Variations: UIT’s absolute OOD EM gains persist across highly divergent unified templates (10.9 pp from raw to UIT in FLAN-style zero-shot testing).
• Model Scaling: All model scales (T5-small to T5-XXL; $\mathcal{F}_s$1M to $\mathcal{F}_s$2B) benefit equally from format unification ($\mathcal{F}_s$3 pp OOD EM improvements).
• Task Diversity vs. Format Consistency: Without unification, task diversity may harm generalization; with UIT, combining task diversity and format consistency yields best results (e.g., $\mathcal{F}_s$4 vs $\mathcal{F}_s$5 EM for src+both vs src+same/diff).
• Ablation Studies (SCAR): Removing quality constraints leads to selection of stylistically consistent but low-quality data, significantly harming SFT outcomes; omitting triplet loss or referenced examples undermines generalization, especially in open-ended domains (Liang et al., 2023, Li et al., 2024).

6. Practical Recommendations and Implications

Key empirical findings and guidelines:

1. Format or style inconsistency alone can cause catastrophic OOD generalization failures (EM $\mathcal{F}_s$6 0–5%).
2. Automatic format transfer using $\mathcal{F}_s$7–10 parallel seeds, LLM-powered conversion, and local perplexity-based denoising offers scalable, annotation-efficient unification.
3. Distilled local format mappers (e.g., GPT-J) closely approach expensive API-based transfer post-training.
4. For any multi-source instruction-tuning project,
   • Select and standardize on a unified template format $\mathcal{F}_s$8
   • Collect seed parallel mappings
   • Run LLM-based conversion with candidate perplexity filtering (e.g., via a small LM)
   • Fine-tune on the style- and format-unified dataset
5. When selecting data for SFT, explicitly rank by presentation and creativity consistency to dramatically reduce required data size without sacrificing (often improving) final LLM performance.

These findings jointly support the position that consistency—in both surface form and content uncertainty—is a dominant determinant of robust instruction-tuning, and that format/style-unification methodologies such as UIT and SCAR operationalize this for practical pipeline optimization (Liang et al., 2023, Li et al., 2024).