Pattern-Aware LLM Rationale Annotators

• The paper demonstrates that pattern-aware synthetic rationales can match or exceed human annotation performance with minimal labeled data.
• It introduces a two-stage SFT+RLVR pipeline that enforces stable reasoning patterns using patterned prompts.
• Empirical results on tasks like NSM and TPC show efficiency gains and performance parity with large-scale human annotations.

Pattern-Aware LLMs as Rationale Annotators (PARO) are a class of frameworks and prompting methodologies that enable LLMs to generate synthetic, procedurally-aligned rationales for downstream tasks—substantially reducing the need for costly, large-scale human annotation while maintaining or surpassing reasoning performance. PARO’s core innovation is the explicit encoding and enforcement of stable “reasoning patterns” through pattern-based prompts presented to potent LLMs, turning them into effective and scalable rationale annotators without major architectural or training loop modifications.

1. Formal Definition of Reasoning Patterns

A reasoning pattern, denoted $\mathcal{P}$, is an abstract, fixed procedural template capturing a shared solution strategy for a broad class of tasks—termed “patterned reasoning tasks.” Each instance $i$ of such a task introduces specific content $\mathcal{C}_i$ (e.g., factual data, numbers, text spans). The solution for $i$ is produced by a deterministic or stochastic “pattern execution” function:

$f(\mathcal{P}, \mathcal{C}_i) \rightarrow y_i$

where $y_i$ is the correct output. While instance content $\mathcal{C}_i$ varies widely, the underlying pattern $\mathcal{P}$ remains invariant. For example, in Numerical Semantic Matching (NSM), $\mathcal{P}_{\rm NSM}$ comprises “(1) ground numbers, (2) interpret their semantic frame, (3) align entities, (4) decide equivalence.” In Transaction-Purpose Classification (TPC), the template consists of “(1) identify holder & direction, (2) extract cues, (3) apply taxonomy rules, (4) finalize category.” This pattern invariance underlies the viability of synthetic rationale generation in PARO frameworks (Pang et al., 14 Oct 2025).

2. Algorithmic Structure and Workflow

PARO maintains the widely adopted two-stage Supervised Fine-Tuning plus Reinforcement Learning with Verifiable Rewards (SFT+RLVR) pipeline. Its novelty lies in substituting human-authored rationales with “pattern-aware” synthetic rationales generated by high-capacity LLMs under tightly guided prompts.

(a) Prompting the Rationale Annotator

A performant LLM (e.g., Qwen3-235B-A22B-Thinking) receives a prompt that: (i) restates task instructions, (ii) enumerates procedural steps of $\mathcal{P}$ in natural language, and (iii) includes two hand-crafted rationale exemplars. Crucially, the gold answer is not provided, requiring the model to execute the pattern and produce a valid chain-of-thought culminating in the correct answer. Example textual prompt for NSM:

$i$1

Applied to each (question, answer) pair in the unlabeled dataset, this produces $i$0.

(b) SFT with Synthetic Rationales

The model $i$1 is fine-tuned by maximizing the likelihood of (rationale, answer) given the question:

$i$2

where $i$3 denotes the serialized rationale–answer string. Empirically: 2 epochs, learning rate $i$4, batch size $i$5, cosine schedule with 1% warmup, gradient clip 1.0.

(c) RLVR Optimization

The fine-tuned model $i$6 is further improved via RLVR on a larger question–answer corpus $i$7:

$i$8

$i$9

with $\mathcal{C}_i$0 iff the extracted answer matches $\mathcal{C}_i$1. Typical RLVR hyperparameters: PPO/VERL, learning rate $\mathcal{C}_i$2, batch size 192, temperature 1.0, 16 rollouts. No additional loss terms are introduced for PARO; its contribution is in rationale curation.

3. Empirical Validation and Performance

PARO’s effect is demonstrated on representative patterned tasks. In NSM:

• Datasets: 110k NSM pairs (544 annual reports); RatQA-10k (10k human-annotated), DirQA-100k (unannotated). Test: 20k each from annual reports and IPO prospectuses.
• Causal evidence (Table 1): SFT+RLVR on all 10k human rationales $\mathcal{C}_i$3 90.3% accuracy, F1 78.4; using only 1k human rationales $\mathcal{C}_i$4 89.8%/77.2; corrupting 25% of rationales does not meaningfully degrade F1. This demonstrates pattern preservation—not volume or gold quality—is determinative (Pang et al., 14 Oct 2025).
• Behavioral evidence: SFT+RLVR models display signature tokens (“different,” “consistent,” “year,” “income”) aligned with NSM pattern steps; contrast: generic connectors dominate in non-pattern-aligned baselines.
• Human vs. PARO (Table 2): SFT(1k, PARO)+RLVR: 92.2% accuracy, F1 83.6; SFT(10k, Human)+RLVR: 92.3%/83.2. On TPC (1k), PARO marginally outperforms human rationales (accuracy 88.2 vs 87.9; F1 87.9 vs 87.2). 1k synthetic rationales sufficed to match/beat 10k human rationales.

This data indicates that, for patterned reasoning tasks, pattern-aligned synthetic rationales enable exceptional efficiency and performance parity or improvement relative to large-scale human annotation.

4. Implementation Details and Hyperparameters

Key training and model parameters utilized in PARO frameworks are as follows:

Component	Model/Hyperparameters	Notes
SFT Backbone	Qwen3-8B
SFT Regimen	2 epochs, lr $\mathcal{C}_i$5, batch 20/GPU, context 4096/1024, cosine warmup 1%, gradient clip 1.0	24× H100 GPUs
RLVR Regimen	PPO/VERL, lr $\mathcal{C}_i$6, batch 192, temp 1.0, 16 rollouts	Full question–answer data
PARO Annotator LLM	Qwen3-235B-A22B-Thinking	Two-shot, pattern-annotated prompt

Ablation studies confirm that explicit pattern listing in prompts constitutes the only additional supervision; no architecture or objective modification is necessary.

5. Extensions: Pattern-Aware LLMs in Machine Translation Quality Estimation

PARO methodologies have been extended to domains such as Machine Translation Quality Estimation (MTQE), where LLMs synthesize segment-level MQM-style annotations to train automatic QE models. The Prompt-Pattern-based-MQM (PPbMQM) protocol compels LLMs (e.g., GPT-4o) to generate MQM-labeled data using patterned prompts with explicit top-level error categories (e.g., Accuracy, Fluency, Style, Terminology, Locale Convention), a 1–5 severity scale, and JSON-structured output. Key experimental results (Wang et al., 11 Mar 2026):

• Segment-level QE models trained on PPbMQM-annotated data achieved segment-level Pearson’s $\mathcal{C}_i$7 (Zh–En), exceeding models trained on gold human MQM ($\mathcal{C}_i$8). In low-quality buckets, PPbMQM-based models were superior ($\mathcal{C}_i$9 vs. $i$0).
• Training protocols: COMET-QE, default hyperparameters, AdamW, batch size 16, epochs 3.
• Few-shot prompt enhancements (category definitions, in-prompt examples) improve annotation consistency and metric alignment.

This extension demonstrates the viability of pattern-aware rationale annotation for semantic quality tasks beyond the original procedural reasoning domain.

6. Limitations and Prospective Research

PARO is effective on patterned reasoning tasks but exhibits limitations when reasoning strategies cannot be easily formalized—such as open-ended mathematical reasoning or tasks with substantial strategy adaptation across instances. Shortcomings can arise from prompt pattern mis-specification, or LLMs hallucinating irrelevant reasoning steps due to vague templates. A plausible implication is decreased reliability on tasks lacking stable procedural groundings.

Future directions include:

• Automatic pattern discovery from unlabeled instances.
• Hybrid human–LLM supervision, blending pattern enforcement with in-the-loop corrections.
• Application to domains with complex, nested patterns (temporal, spatial, logical).

In the broader context, PARO recasts the human role from rationale annotator to designer of concise pattern prompts, significantly reducing manual labeling costs while retaining SFT+RLVR effectiveness (Pang et al., 14 Oct 2025, Wang et al., 11 Mar 2026).