FG-PAN: Fine-Grained Patch Alignment

• The paper presents a novel zero-shot framework that aligns visual patch embeddings with LLM-generated textual prototypes for fine-grained brain tumor subtype classification.
• It introduces a local feature refinement module using window attention and gated feature fusion to contextualize patch-level evidence.
• Empirical results show consistent improvements of up to 10% in balanced accuracy, F1-score, and AUROC across multiple brain tumor benchmarks.

Searching arXiv for the primary FG-PAN paper and closely related alignment papers for citation support. Fine-Grained Patch Alignment Network (FG-PAN) denotes a model family centered on patch-level alignment, but the term is used most explicitly for a zero-shot digital pathology framework introduced in "Enhancing Zero-Shot Brain Tumor Subtype Classification via Fine-Grained Patch-Text Alignment" (Gan et al., 3 Aug 2025). In that formulation, FG-PAN is designed for fine-grained brain tumor subtype classification from histopathological whole slide images under scarce annotation, and it combines a local feature refinement module with a fine-grained text description generation module to align visual patch embeddings with pathology-aware semantic prototypes (Gan et al., 3 Aug 2025). The same acronym can also be used more loosely for patch-based alignment strategies in landmark localization, particularly the multi-scale local patch regression pipeline of "Coarse-to-fine Face Alignment with Multi-Scale Local Patch Regression" (Huang et al., 2015). By contrast, the pose-aligned representation work "Aligned to the Object, not to the Image: A Unified Pose-aligned Representation for Fine-grained Recognition" addresses alignment at the level of object pose and hierarchical regions rather than a model explicitly named FG-PAN (Guo et al., 2018).

1. Terminological scope and problem setting

In its explicit contemporary usage, FG-PAN is a "novel zero-shot framework tailored for digital pathology" that targets the fine-grained classification of brain tumor subtypes from histopathological whole slide images (Gan et al., 3 Aug 2025). The motivating problem is that subtle morphological variations and the scarcity of annotated data limit the effectiveness of standard vision-LLMs, whose fine-grained pathological discrimination remains suboptimal (Gan et al., 3 Aug 2025). FG-PAN is therefore framed around patch-text alignment rather than slide-level template matching alone.

The acronym also admits a broader descriptive use for architectures that refine predictions through localized patch processing. The face alignment pipeline in (Huang et al., 2015) is described as a coarse-to-fine system in which a global convolutional network first predicts landmarks from a holistic facial image, after which a pyramid of multi-scale local image patches is cropped for each landmark and passed through a refinement network repeatedly until convergence. In that sense, the description characterizes a patch alignment network, but this is an interpretive extension rather than the canonical named model. A plausible implication is that "FG-PAN" names two distinct alignment traditions: one in digital pathology based on cross-modal patch-text matching, and another in landmark localization based on iterative local patch regression.

The object-centric representation in (Guo et al., 2018) constitutes a related but separate lineage. That paper proposes pose-aligned patches and a unified object representation built from a hierarchy of pose-aligned regions to disentangle pose and appearance in fine-grained recognition. This suggests a conceptual affinity—alignment to semantically meaningful structure rather than raw image axes—but not identity of architecture or nomenclature.

2. Core architecture in zero-shot digital pathology

FG-PAN in (Gan et al., 3 Aug 2025) is presented as a plug-and-play module that can be "dropped on top" of any frozen pathology foundation model, with examples including Conch v1.5, UNI, and Virchow. Its architecture contains two parallel streams.

The visual stream first selects a small set of representative patches $\{x_1,\dots,x_m\}$ from a whole-slide image, extracts patch embeddings $f_i$ via a frozen backbone $\epsilon$, and refines those embeddings with Local Window Attention and Gated Feature Fusion to obtain $H_i \in \mathbb{R}^d$ (Gan et al., 3 Aug 2025). The textual stream uses a LLM, specifically DeepSeek-R1, to generate fine-grained, pathology-aware textual descriptions for each tumor subtype $c \in \{1 \dots C\}$, and then embeds those descriptions into semantic prototypes $T_c \in \mathbb{R}^d$ through the frozen text encoder (Gan et al., 3 Aug 2025).

Patch-text alignment is then performed by cosine similarity with a learnable temperature $\tau$, producing patch-level class scores that are aggregated into a slide-level prediction through coordinate-aware weighting (Gan et al., 3 Aug 2025). The formulation is given as

$$f_i = \epsilon(x_i) \in \mathbb{R}^d, \quad i=1 \dots N$$

$$\{H_i\} = \mathrm{REFINE}(\{f_i\}), \qquad \{T_c\} = \mathrm{ENC\_TEXT}(\{\mathrm{prompt}_c\})$$

$$s_i^c = \frac{H_i^\top T_c}{\|H_i\|\|T_c\|}, \qquad p_i^c = \frac{\exp(s_i^c/\tau)}{\sum_k \exp(s_i^k/\tau)}.$$

This architecture is explicitly zero-shot. The semantic side is constructed from language-generated class descriptions rather than supervision-specific classifier weights, and the visual side refines local evidence prior to alignment. The design therefore couples spatial modeling within the slide to semantic specialization in text space.

3. Local feature refinement and representative patch processing

The local feature refinement module in (Gan et al., 3 Aug 2025) begins with WSI preprocessing. Whole slide images are first filtered by simple threshold-based tissue segmentation. From the tissue mask, a fixed budget of 200–300 patches is sampled per WSI, discarding blank or benign regions, and a small set of $f_i$0 representative patches is then chosen; although farthest-point sampling or importance scores are mentioned as possibilities, the implementation uses stratified random sampling guided by the CLAM toolbox (Gan et al., 3 Aug 2025).

Local Window Attention operates on the selected patches with explicit spatial coordinates $f_i$1. The patches are divided into non-overlapping windows $f_i$2 of size $f_i$3, with

$f_i$4

For each window, patch features are stacked as

$f_i$5

then projected to queries, keys, and values,

$f_i$6

where $f_i$7 (Gan et al., 3 Aug 2025). Relative-positional self-attention within each window is computed by

$f_i$8

with $f_i$9 a learned relative bias (Gan et al., 3 Aug 2025).

The attention outputs are then processed by Gated Feature Fusion. For each patch $\epsilon$0, the model takes head outputs $\epsilon$1 and computes head-wise gates with a sigmoid:

$\epsilon$2

where $\epsilon$3 and $\epsilon$4, followed by

$\epsilon$5

The fused representation is

$\epsilon$6

with $\epsilon$7 and $\epsilon$8 (Gan et al., 3 Aug 2025).

The stated role of Local Window Attention and Gated Feature Fusion is to yield refined patch embeddings $\epsilon$9. This suggests that the architecture treats fine-grained pathology recognition as a problem of contextualized local evidence rather than isolated patch classification.

4. Fine-grained textual prototypes and patch-text alignment

The semantic module in (Gan et al., 3 Aug 2025) generates class descriptions by prompting DeepSeek-R1 without fine-tuning the LLM. The prompt template is:

“As a neuropathology expert, what distinctive multiscale features on whole slide images differentiate [class] from other brain tumors per WHO CNS5 criteria? Generate discriminative attribute pairs combining molecular profiles and histopathological signatures using the format: ‘[class] with [molecular feature] and [histological pattern].’”

An example description reported in the data is:

“Anaplastic oligodendroglioma, IDH-mutant and 1p/19q-codeleted with IDH1 R132H mutation and microcystic honeycomb architecture containing fried-egg cells.”

These natural-language descriptions are encoded by the frozen VLM text encoder to produce semantic prototypes $H_i \in \mathbb{R}^d$0 (Gan et al., 3 Aug 2025). The alignment mechanism uses cosine similarity,

$H_i \in \mathbb{R}^d$1

learnable temperature scaling $H_i \in \mathbb{R}^d$2, and patch-level softmax

$H_i \in \mathbb{R}^d$3

Training beyond the patch extractor and text encoder is end-to-end with a patch-level cross-entropy objective,

$H_i \in \mathbb{R}^d$4

where $H_i \in \mathbb{R}^d$5 is the ground-truth patch label; in zero-shot training, global slide labels are proxied to all informative patches (Gan et al., 3 Aug 2025). The paper explicitly states that there is no additional margin loss or global contrastive objective.

This module is central to FG-PAN’s claim that class separability can be increased in both visual and semantic spaces by aligning refined visual features with LLM-generated fine-grained descriptions (Gan et al., 3 Aug 2025). A plausible implication is that the architecture uses language not merely for initialization or prompt engineering, but as a structured source of subtype-specific morphological priors.

5. Training configuration, datasets, and empirical results

The training and test regime in (Gan et al., 3 Aug 2025) is specified around zero-shot transfer. Training uses 500 WSIs from CAMELYON, described as "500 sentinel lymph node slides ‑ irrelevant to brain tumors," split into approximately 200–300 patches each. Testing is performed zero-shot on EBRAINS, IPD-Brain, BRACS, and TCGA-Brain:

Dataset	Reported scope
CAMELYON	500 WSIs for training
EBRAINS	1,200 brain-tumor WSIs covering 30 fine-grained subtypes, with 12 coarse subtype grouping
IPD-Brain	484 WSIs, 3 glioma subtypes
BRACS	547 breast WSIs, 7 classes
TCGA-Brain	429 WSIs, binary GBM vs LGG

Preprocessing is summarized as tissue segmentation plus CLAM tiling, followed by 200–300 candidate patches per patient and representative patch selection (Gan et al., 3 Aug 2025). The feature extractor $H_i \in \mathbb{R}^d$6 is frozen Conch v1.5, or an alternative pathology foundation model. Optimization uses AdamW with learning rate $H_i \in \mathbb{R}^d$7, weight decay $H_i \in \mathbb{R}^d$8, batch size of 4 WSIs, 20K iterations total, and early stopping if validation loss plateaus. Temperature $H_i \in \mathbb{R}^d$9 and gating parameters are initialized to 1 and 0.5 respectively, and the implementation is in PyTorch on a single RTX 4090 (Gan et al., 3 Aug 2025).

Across five benchmarks—EBRAINS$c \in \{1 \dots C\}$0, EBRAINS$c \in \{1 \dots C\}$1, IPD-Brain$c \in \{1 \dots C\}$2, BRACS$c \in \{1 \dots C\}$3, and TCGA$c \in \{1 \dots C\}$4—FG-PAN is reported to "consistently raise balanced accuracy, F1-score and AUROC of six backbone models (UNI, Conch, Conch v1.5, Virchow, PLIP, CTransPath)" (Gan et al., 3 Aug 2025). A specific snapshot on EBRAINS$c \in \{1 \dots C\}$5 with Conch v1.5 is:

Metric	Baseline	+FG-PAN
Balanced accuracy	0.493	0.572
F1	0.542	0.597
AUROC	0.913	0.923

Balanced accuracy improvements on EBRAINS$c \in \{1 \dots C\}$6 are also listed for several backbones: UNI 0.212→0.228, Conch 0.357→0.405, Conch v1.5 0.493→0.572, Virchow 0.412→0.465, PLIP 0.247→0.285, and CTransPath 0.205→0.224 (Gan et al., 3 Aug 2025). Applying FG-PAN on CHIEF, Prov-GigaPath, and UNI2-H is said to yield +3–5% accuracy improvements.

Ablations reported in Tables 3–4 separate the contribution of the semantic and spatial components. Removing both LWA+GFF and FTDG yields the baseline; "+FTDG only" gives approximately +4–5% balanced accuracy, "+LWA+GFF only" gives approximately +5–6%, and using both gives the largest gain, approximately +7–10% (Gan et al., 3 Aug 2025). Textual prompt quality is further assessed by comparing different LLMs, with average inter-class distances of approximately 1.66–1.70 for generated descriptions versus 1.91 using class names alone, and this is reported to correlate with improved zero-shot accuracy (Gan et al., 3 Aug 2025).

6. Relationship to patch-based alignment in face alignment and fine-grained recognition

The face alignment system of (Huang et al., 2015) provides a different, earlier patch-based alignment paradigm. It applies a global convolutional network to a holistic facial image to obtain an initial landmark prediction, then crops a pyramid of multi-scale local image patches for each landmark and feeds them to a new network to refine the prediction; the process can be repeated several times until convergence (Huang et al., 2015). The mathematical description in the supplied data writes the global estimate as

$c \in \{1 \dots C\}$7

followed by iterative residual updates

$c \in \{1 \dots C\}$8

The losses are pure coordinate-regression losses:

$c \in \{1 \dots C\}$9

and

$T_c \in \mathbb{R}^d$0

On the 300-W dataset, the reported normalized errors are 3.83% on the common subset, 7.46% on the challenging subset, and 4.54% on the fullset, outperforming prior work such as CFSS and DCR (Huang et al., 2015).

This earlier formulation differs fundamentally from the pathology FG-PAN in objective, modality, and supervision. The face model is a supervised coordinate-regression cascade over facial landmarks, whereas the pathology model is a zero-shot patch-text alignment framework over subtype semantics. The commonality lies in the use of localized patch evidence to correct or sharpen a coarse representation.

The pose-aligned fine-grained recognition model in (Guo et al., 2018) offers another relevant comparison. That work argues that dramatic appearance variation due to pose challenges fine-grained recognition, proposes a unified object representation built from a hierarchy of pose-aligned regions, and reports improvements of nearly 2% on CUB-200 and more than 8% on NABirds (Guo et al., 2018). Rather than image-axis-aligned regions, it uses pose-aligned patches whose features are robust to pose, scale, and rotation. This suggests a broader alignment principle shared with FG-PAN: recognition improves when appearance is represented relative to semantically meaningful structure instead of raw image layout.

7. Interpretation, limitations, and common points of confusion

A common source of confusion is that "FG-PAN" does not denote a single universally established architecture across domains. In the supplied literature, the explicit named model is the digital pathology framework of (Gan et al., 3 Aug 2025). The face alignment paper (Huang et al., 2015) is described in the data as realizing "what one could call a Fine-Grained Patch Alignment Network," which indicates an analogical label rather than the original published model name. The object-pose paper (Guo et al., 2018) is still further removed, since it addresses pose-aligned representation and not an FG-PAN model per se.

Within the pathology setting, the paper identifies several limitations and future directions. Patch selection remains heuristic, prompt generation is LLM-dependent and requires manual vetting, and current training uses proxy patch labels from slide labels rather than explicit weak supervision at patch level such as MIL (Gan et al., 3 Aug 2025). The paper suggests that dynamic, attention-based selection, closed-loop prompt tuning, multimodal LLMs, and explicit weak supervision at patch level could further refine the framework.

The reported robustness results indicate that FG-PAN improves zero-shot performance not only on brain-tumor benchmarks but also on out-of-domain BRACS and IPD-Brain, which the paper attributes to two effects: spatial refinement combats fragmentation of morphological cues, and pathology-aware prompts supply domain knowledge missing in generic templates (Gan et al., 3 Aug 2025). This suggests that the central contribution of FG-PAN is not merely architectural modularity, but a specific synthesis of contextual patch modeling and semantically enriched textual prototypes for fine-grained classification under distribution shift and annotation scarcity.