Visual Self-Refinement for Autoregressive Models

• The paper introduces a paradigm that integrates instance-specific radiomic feedback to iteratively refine AR model prompts, significantly improving lung nodule classification accuracy.
• The methodology employs a transformer-based architecture that processes images and text in a unified causal sequence to enable real-time context adaptation and robust cross-modal alignment.
• Empirical evaluations demonstrate increased recall, F1 scores, and ROC AUC, particularly in ambiguous cases, underscoring the clinical potential of feedback-driven self-refinement.

Visual self-refinement for autoregressive models is a paradigm for enhancing vision-centric generative or discriminative models by incorporating feedback mechanisms, prompt adaptation, or learned context adjustment based on visual input statistics or dynamic instance features. In medical imaging and computational radiology, it encompasses strategies where autoregressive (AR) vision or vision-LLMs iteratively update their prompts, context, or embeddings using high-dimensional information—such as radiomics—to improve prediction accuracy, discrimination, and robustness, especially in complex or uncertain cases.

1. Core Principles of Visual Self-Refinement in Autoregressive Models

Visual self-refinement refers to methods that allow AR models—particularly those operating on images or mixed modalities—to modify their internal representations, prompts, or embeddings based on real-time visual input characteristics or derived feature vectors. This stands in contrast to fixed-context or static-prompt AR models, embedding dynamic, input-conditioned self-adaptation into the model pipeline. In medical vision-language tasks, this often merges hand-crafted descriptors (e.g., radiomic features) with autoregressive visual encoders and prompt-tuning architectures to tightly couple pixel-level input characteristics with downstream prediction or generation objectives (Khademi et al., 26 Mar 2025).

In such frameworks, the AR decoder (e.g., transformer-based as in AIMv2) processes both images tokenized into patches and text tokens as a single causal sequence, enabling complex cross-modal dependencies. Visual self-refinement then operates by conditioning the tokenization, prompting, or fusion process on instance-aware computations—such as learned deltas from radiomic vectors—improving cross-modal alignment and increasing prediction accuracy, most notably where traditional, static methods underperform.

2. Architectural Implementations: Example in AutoRad-Lung

A concrete visual self-refinement pipeline is exemplified by AutoRad-Lung—a radiomic-guided, prompting AR vision-LLM for lung nodule malignancy classification (Khademi et al., 26 Mar 2025). The framework integrates:

• A large-scale AR vision encoder (AIMv2) that processes CT slices as sequences of non-overlapping patches, with causal modeling jointly over image and text.
• Conditional prompt adaptation via a Meta-Net: Given high-dimensional radiomic vectors ($r_j \in \mathbb{R}^{N_r}$), a neural meta-network computes an instance-specific delta token:

$$\delta_j = h_\phi(r_j) = W_2\,\mathrm{ReLU}(W_1\,r_j + b_1) + b_2$$

• Instance-aware context construction adjusts static context embeddings $\{v_m\}$ with $\delta_j$:

$v_m(r_j) = v_m + \delta_j$

• Full prompts for each class $c_i$:

$t_i(r_j) = [v_1(r_j), \ldots, v_M(r_j), c_i]$

These are encoded by a transformer-based text backbone.

At inference, each CT volume is segmented and radiomic features extracted. These guide prompt construction, aligning the AR vision encoder's output tightly with relevant, case-based clinical descriptors. Classification is performed by computing the similarity between AR-encoded images and these context-adapted prompts.

3. Model Training Objectives and Workflow

Visual self-refinement in AR models employs both autoregressive pre-training and prompt-adaptive fine-tuning:

• Pre-training: The AR vision-language backbone is trained to predict the next token (patch or subword), leveraging large multimodal datasets. Objective:

$$\mathcal{L}_{\mathrm{AIM}} = - \sum_{k=1}^{L+T} \log p(x_k \mid x_{1:k-1})$$

• Fine-tuning: The task-specific head uses input-adaptive prompts, optimizing a cross-entropy loss over classes:

$$p(y_j = i \mid I_j, r_j) = \frac{\exp(\mathrm{sim}(x_j, z_{i,j})/\tau)}{\sum_{k}\exp(\mathrm{sim}(x_j, z_{k,j})/\tau)}$$

$$\mathcal{L}_{\mathrm{CE}} = - \sum_j \log p(y_j \mid I_j, r_j)$$

All prompt and context parameters, and optionally the vision encoder, are updated using SGD with momentum, coordinated with cosine learning rate decay and weight decay regularization.

Inference strictly follows the same instance-conditioned prompt pathway, guaranteeing that the conditioning provided by radiomic vectors is directly utilized.

4. Empirical Evaluation and Benchmarks

Visual self-refinement AR models have demonstrated superior performance over static AR or parallel deep learning methods in lung nodule malignancy discrimination. In the LIDC-IDRI evaluation, AutoRad-Lung achieves:

Method	Accuracy	Benign Recall/F1	Malignant Recall/F1	Unsure Recall/F1
CLIP-Lung	60.9±0.4	67.5/64.4	60.9/66.3	53.4/54.1
AIMv2	58.5±0.3	62.5/55.2	43.6/45.6	51.3/52.3
AutoRad-Lung	64.6±1.7	75.3/69.5	65.6/60.6	62.3/71.6

Notably, AutoRad-Lung delivers increased recall and F1 in the "unsure" class (often the most ambiguous for clinicians), as well as an overall +3.7% absolute accuracy increase over the best prior model. The ROC AUC exceeds 0.85 for each class (Khademi et al., 26 Mar 2025).

This suggests that feedback-driven prompt conditioning—i.e., self-refinement from visual-derived feature spaces—enables discrimination advantages, specifically in ambiguous or edge-case predictions.

5. Relation to Broader Medical Vision and Deep Learning

Visual self-refinement represents a progression from fixed deep CNN classifiers (e.g., DeepLung's 3D dual-path network plus gradient boosting (Zhu et al., 2017)) and even supervised AR image models toward contextually adaptive, feedback-driven strategies.

While high-performing 3D CNNs and gradient boosting have reached radiologist-level performance in detection and binary classification (FROC=83.4%, nodule-level accuracy=90.4%, kappa=0.63 (Zhu et al., 2017)), static context and feature selection can limit robustness in visually ambiguous settings. Integrating dense, quantitative radiomic signatures and using them to modulate AR context is positioned to address these limitations, especially in tasks requiring nuanced, instance-sensitive discrimination.

A plausible implication is that such self-refinement pipelines can generalize to other medical vision tasks, either extending to full-3D networks or adapting to resource-limited imaging modalities.

6. Clinical Significance, Limitations, and Future Directions

Visual self-refinement in AR vision-LLMs removes dependence on subjective or unavailable radiologist annotations by relying solely on quantitative features extracted from the imaging data. In AutoRad-Lung, handcrafted radiomics guide both train and test-time prompts, ensuring clinical validity and case specificity. This reduces the bias introduced by fixed, human-defined prompts and offers improved generalization across patient cohorts.

Limitations include:

• Preprocessing overhead and reliance on robust segmentation for radiomics extraction.
• The use of 2D central slices, with prospective gains from full 3D extensions.
• Risk of overfitting when jointly fine-tuning all parameters on limited labeled datasets.
• Requirement for further clinical validation and workflow integration.

Further directions include extending self-refinement to end-to-end 3D models, optimizing regularization for medical-scale data, and broader evaluation across multi-institutional datasets (Khademi et al., 26 Mar 2025). The demonstrated improvements in difficult-to-classify cases and recall across all malignancy classes indicate that visual self-refinement is a critical frontier in medical AR modeling.