Fine-R1: Fine-Grained Recognition

• The paper introduces Fine-R1, a training framework that integrates chain-of-thought supervision with R1-style reinforcement to boost fine-grained visual recognition.
• It employs a two-stage training approach, combining supervised fine-tuning on a curated CoT dataset with Triplet Augmented Policy Optimization to mitigate data scarcity and overfitting.
• Empirical results across six benchmarks demonstrate significant gains in seen and unseen category accuracy, highlighting its robust generalization in open- and closed-world settings.

Fine-R1 is a training framework and resulting multi-modal LLM for fine-grained visual recognition (FGVR) that adapts a general vision-LLM into a strong open-world classifier through explicit chain-of-thought reasoning and an R1-style reinforcement-learning stage tailored to fine-grained categories such as bird species, car models, aircraft variants, and dog breeds. It is designed to address two persistent difficulties in FGVR: the need for large amounts of annotated data when adapting general-purpose multimodal LLMs, and weak generalization to unseen sub-categories. In the reported setting, Fine-R1 is trained with only 4 labeled images per seen sub-category and is evaluated in both closed-world and open-world regimes across six FGVR benchmarks (He et al., 7 Feb 2026).

1. Problem formulation and scope

Fine-grained visual recognition differs from coarse-grained recognition in that the target labels are highly similar sub-categories rather than broad semantic classes. The paper frames this as a setting where general MLLMs are good at broad recognition but much weaker at distinguishing closely related sub-categories, and where standard supervised adaptation tends to overfit to seen labels and generalize poorly to unseen ones. Fine-R1 is proposed specifically to exploit the model’s intrinsic world knowledge and language reasoning for FGVR rather than treating the task as a purely discriminative classification problem (He et al., 7 Feb 2026).

The framework distinguishes two inference regimes. In the open-world setting, the model receives an image $x$ and a query $q$ such as “What is the name of the bird in the photo?” and generates a free-form textual label $c \in \mathcal{T}$. In the closed-world setting, the query includes a candidate list $C$, effectively turning recognition into multiple-choice selection. The underlying MLLM is treated as a conditional text generator,

$$f_{\mathrm{MLLM}} : \mathcal{X} \times \mathcal{T} \to \mathcal{T},$$

with no task-specific classifier head added (He et al., 7 Feb 2026).

The evaluation protocol is a unified base-to-new split across six datasets: CUB-200-2011, Stanford Cars-196, Stanford Dogs-120, Oxford Flowers-102, Oxford-IIIT Pet-37, and FGVC-Aircraft. For each dataset, 60% of sub-categories are designated seen and 40% unseen, and a single Fine-R1 model is trained jointly across all six datasets using only 4 labeled images per seen sub-category (He et al., 7 Feb 2026).

2. Base model and chain-of-thought supervised fine-tuning

Fine-R1 is not a new architecture. It is a training recipe applied to Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct. The model retains the underlying Qwen2.5-VL components—a vision encoder, a LLM, and the existing multimodal fusion module—and specialization is introduced entirely through training and output formatting rather than through new architectural modules (He et al., 7 Feb 2026).

The first stage is Chain-of-Thought Supervised Fine-Tuning (CoT SFT). Its central design is a four-part FGVR reasoning pattern:

1. visual analysis,
2. candidate sub-categories,
3. detailed comparison,
4. final prediction.

During data construction, these stages are enclosed in tags such as <analysis>, <options>, <comparison>, and <prediction>. At inference, the model is prompted to emit reasoning inside > ... and the final answer inside <answer> ... </answer>. The intent is to emulate expert identification behavior: describe discriminative attributes, enumerate plausible fine-grained candidates within the same super-category, compare them using visual and textual knowledge, and then commit to a label (He et al., 7 Feb 2026).

The CoT dataset is deliberately small and curated. One image per sub-category is sampled from the FGVR datasets, and Qwen2.5-VL-32B is used with Image-level Visual Concept Selection to harvest concise discriminative visual concepts. A structured prompt then asks for a full FGVR reasoning chain, and only outputs whose final predicted sub-category exactly matches ground truth are retained. After filtering mixed-language responses and manually verifying that the predicted sub-category appears in the candidate list and matches the ground truth, the final CoT dataset contains 404 high-quality samples (He et al., 7 Feb 2026).

The SFT objective is standard teacher-forced next-token prediction over the complete reasoning-and-answer sequence: $$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\sum_{t=1}^{T} \log p_\theta(y_t \mid y_{<t}, x, q).$$ This stage converts the base Qwen model into what the paper describes as a strong open-world classifier, but it does not by itself solve unseen-category generalization; the paper explicitly reports that pure CoT SFT still overfits to seen classes and reduces unseen-category closed-world accuracy relative to the zero-shot baseline (He et al., 7 Feb 2026).

3. Triplet Augmented Policy Optimization

The second stage is Triplet Augmented Policy Optimization (TAPO), an R1-style reinforcement-learning procedure built on DAPO-like policy optimization. TAPO is organized around a triplet

$T = (x, x_{\mathrm{pos}}, x_{\mathrm{neg}}),$

where $x$ is an anchor image, $x_{\mathrm{pos}}$ is an image from the same sub-category, and $x_{\mathrm{neg}}$ is an image from the most visually similar but different sub-category (He et al., 7 Feb 2026).

The paper separates TAPO into two mechanisms. Intra-class Augmentation addresses high intra-class variance. Rollouts are sampled under the old policy from both the anchor and the positive image, their rewards are pooled, and policy updates are applied only under the anchor conditioning. The stated effect is to force the model to exploit information that is invariant across different appearances of the same class. Empirically, the best performance occurs around a $q$0 anchor-to-positive rollout ratio, which the paper interprets as evidence that the gain comes from diversity rather than merely from more anchor samples (He et al., 7 Feb 2026).

Inter-class Augmentation addresses low inter-class variance. For a token sequence $q$1, the paper defines

$q$2

and introduces a KL-based objective that increases the distinction between the model’s response distributions under matched and mismatched images. Double entropy regularization is added to stabilize this contrastive pressure. In effect, the policy is trained not only to make correct predictions for the anchor class but also to sharply downweight the same reasoning-and-answer trajectory when the image is replaced with a confusing near-neighbor class (He et al., 7 Feb 2026).

The full TAPO objective combines token-level clipped policy optimization, the inter-class KL term, and entropy regularization. The paper’s reported training configuration uses global batch size 128, rollout batch size 384, learning rate $q$3, $q$4 responses per prompt, 10 RL epochs for the 3B model, 5 RL epochs for the 7B model, and 4 NVIDIA A6000 GPUs. The DAPO settings include clipping factors $q$5 and $q$6, removal of the reference-KL term, token-level loss averaging, and dynamic sampling with up to 20 retries (He et al., 7 Feb 2026).

4. Training regime and evaluation methodology

Fine-R1’s training protocol is explicitly two-stage. Stage 1 performs full fine-tuning on the 404-sample CoT dataset for 10 epochs using LLaMA-Factory. Stage 2 performs TAPO on the 4-shot subset of seen categories. The paper emphasizes that the unified model is trained jointly across all six FGVR datasets rather than separately per dataset (He et al., 7 Feb 2026).

Closed-world evaluation uses standard accuracy on seen and unseen categories. Open-world evaluation uses a relative semantic similarity metric based on SigLIP text embeddings: $q$7 where $q$8 is the super-category, $q$9 is the predicted sub-category, $c \in \mathcal{T}$0 is the ground-truth sub-category, and $c \in \mathcal{T}$1 is cosine similarity between SigLIP text embeddings. This metric is intended to preserve graded semantic correctness in open-world naming, where near-miss predictions can still be more informative than the super-category label alone (He et al., 7 Feb 2026).

A central feature of the evaluation setup is that it stresses base-to-new generalization. No unseen-class images are used during RL. The reported results therefore measure whether structured CoT plus TAPO can use few-shot seen-class supervision to improve recognition of both seen and unseen sub-categories, rather than merely memorizing the seen label set (He et al., 7 Feb 2026).

5. Empirical results and ablation findings

On closed-world FGVR, Fine-R1-7B reports average accuracy of 91.71% on seen categories, 85.70% on unseen categories, and 88.71% overall across the six datasets. Relative to Qwen2.5-VL-7B, these correspond to gains of +7.73, +9.28, and +8.51 points. Relative to DeepPerception-7B, the gains are +4.72, +6.46, and +5.59 points. Relative to SigLIP-L, Fine-R1-7B exceeds the reported averages by +3.38 on seen, +5.16 on unseen, and +4.27 overall (He et al., 7 Feb 2026).

On open-world FGVR, Fine-R1-7B reports 82.62% average relative semantic similarity on seen categories, 66.97% on unseen categories, and 74.80% overall. The paper reports a +23.75 point overall gain over Qwen2.5-VL-7B in this setting. Fine-R1-3B reaches 67.32% overall, also substantially above its base model (He et al., 7 Feb 2026).

The ablations clarify the role of each training stage. CoT SFT alone improves seen-category performance but hurts unseen-category closed-world accuracy by 6.12 points relative to the zero-shot baseline, indicating overfitting. A classification-only RL variant does not beat the zero-shot model on unseen classes. “No-Thinking RL” improves seen performance but remains substantially weaker than the full CoT-based pipeline on unseen classes. By contrast, CoT SFT + TAPO yields the strongest base-to-new generalization: in the 3B setting, the reported averages are 88.97% on seen categories, 81.41% on unseen categories, and 85.19% overall, with gains of +7.56 points over SFT on seen categories and +10.05 points over No-Thinking RL on unseen categories (He et al., 7 Feb 2026).

The paper also reports that adding more curated CoT samples improves unseen-category performance rather than overfitting, and that TAPO transfers across backbones: applying TAPO to Qwen2-VL-2B still yields consistent gains over CoT SFT plus plain DAPO. Qualitative examples show Fine-R1 using the intended “visual analysis → candidates → comparison → prediction” structure to disambiguate similar birds, cars, and aircraft by explicitly contrasting subtle attributes such as forehead patches, rooflines, or tail and engine configurations (He et al., 7 Feb 2026).

6. Interpretation, limitations, and broader R1-style context

A recurrent misconception would be to interpret Fine-R1 as a new visual backbone or as evidence that RL substantially changes the underlying visual representation. The paper argues against that reading. On CUB-200 linear probing, Qwen2.5-VL-3B reports 84.26% probe accuracy and Fine-R1-3B 85.00%, which the paper treats as almost no change. It likewise reports minimal differences in taxonomy-aware similarities between species from the same versus different genus. The stated conclusion is that Fine-R1 does not fundamentally change the model’s visual or taxonomic knowledge; it improves the deployment of existing knowledge for FGVR (He et al., 7 Feb 2026).

The paper also notes a dependence on synthetic CoT supervision, since the CoT dataset is generated by Qwen2.5-VL-32B and then filtered and manually checked. This leaves open the possibility that teacher biases shape the downstream reasoning format. A plausible implication is that Fine-R1’s gains are best understood as a strong specialization method for data-scarce FGVR rather than as a universal solution to fine-grained perception (He et al., 7 Feb 2026).

Within the broader multimodal R1-style literature, Fine-R1 belongs to a family of methods that pair structured outputs with reinforcement fine-tuning but adapt the reward and output space to domain-specific goals. VideoChat-R1 applies GRPO with rule-based rewards to spatio-temporal perception tasks such as temporal grounding and tracking, and reports that multi-task reinforcement fine-tuning can improve specialist video tasks without sacrificing general video QA (Li et al., 9 Apr 2025). Weather-R1 adds a logical consistency reward to reduce Self-Contradictory Reasoning in meteorology, explicitly targeting faithfulness between reasoning and final answer in a high-stakes domain (Wu et al., 20 Jan 2026). GenSeg-R1 uses GRPO to train a reason-then-segment pipeline in which a VLM emits boxes and keypoints for a frozen SAM 2 segmenter, extending R1-style optimization to fine-grained referring segmentation (Hegde et al., 10 Feb 2026). PreResQ-R1 adapts GRPO to visual quality assessment with a disentangled reward that jointly optimizes score calibration, ranking, and reasoning coherence (Feng et al., 7 Nov 2025).

Taken together, these comparisons suggest that Fine-R1’s distinctive contribution is not the generic use of reinforcement fine-tuning, but the way it translates the R1-style recipe into the FGVR regime: explicit CoT structured around candidate sub-categories and comparisons, few-shot joint training across six datasets, and triplet-augmented policy optimization that directly targets high intra-class variance and low inter-class variance.