Auto-prompted Generation Model (AGM)

• AGM is an auxiliary segmentation branch that converts preliminary masks into bounding-box prompts for SAM, facilitating automated segmentation under domain shift.
• It integrates shallow-feature uncertainty modeling (SUFM) by combining Gaussian and Poisson perturbations to improve prompt quality on unseen target domains.
• Empirical results on prostate MRI show that adding SUFM and IPEF to AGM boosts Dice scores, confirming its practical benefits in mitigating prompt dependency.

Auto-prompted Generation Model (AGM) denotes, in the FA-SAM framework, an auxiliary trainable segmentation branch that automatically generates bounding-box prompts for the Segment Anything Model (SAM) in single-source domain generalization for medical image segmentation (Zhuo et al., 23 Jul 2025). Its purpose is to remove the dependence of SAM on domain-specific expert-annotated prompts, especially bounding boxes, and thereby enable fully automated segmentation at test time in cross-domain settings where only a single labeled source domain is available during training (Zhuo et al., 23 Jul 2025). Within FA-SAM, AGM operates upstream of SAM: it predicts a preliminary mask from the input medical image, converts that mask into a bounding-box prompt, and supplies the prompt to a frozen SAM prompt encoder, after which a modified SAM decoder produces the final segmentation (Zhuo et al., 23 Jul 2025).

1. Terminological scope and acronym ambiguity

The acronym “AGM” is heavily overloaded in the arXiv literature. In classical numerical analysis and computational mathematics, AGM commonly denotes the arithmetic-geometric mean of Gauss and Legendre, which underlies high-precision algorithms for $\pi$ and elementary functions (Brent, 2018), and also rapid approximations to the period of the simple pendulum (Villarino, 2012). In machine learning, AGM has also been expanded as “Adaptive Generation Model,” a stacking-like ensemble method for tabular classification (Ruan et al., 2020). In logic and belief change, AGM refers to the Alchourrón–Gärdenfors–Makinson framework for belief revision (Falakh et al., 2021).

In the present sense, AGM means Auto-prompted Generation Model and is specific to the FA-SAM framework for medical image segmentation under single-source domain generalization (Zhuo et al., 23 Jul 2025). The term does not designate a general-purpose prompt optimization architecture across arbitrary LLM tasks; rather, it names a component whose output is a SAM-compatible bounding-box prompt derived automatically from a preliminary segmentation mask (Zhuo et al., 23 Jul 2025).

2. Functional position within FA-SAM

FA-SAM is organized around two main innovations: the AGM branch, which includes a Shallow Feature Uncertainty Modeling (SUFM) module, and the Image-Prompt Embedding Fusion (IPEF) module integrated into the SAM mask decoder (Zhuo et al., 23 Jul 2025). Their roles are complementary. AGM is responsible for prompt generation: it replaces manual prompt annotation by predicting a coarse mask and converting that prediction into a bounding box for SAM. IPEF addresses the residual brittleness of SAM when prompts are imperfect by improving fusion between image embeddings and prompt embeddings inside the decoder (Zhuo et al., 23 Jul 2025).

The resulting workflow is sequential rather than end-to-end in the strong differentiable sense described in many prompt-conditioned systems. An input image is first processed by AGM to obtain a preliminary segmentation mask. That mask is then post-processed into a bounding box. The box is passed to the frozen SAM prompt encoder; in parallel, the SAM image encoder provides image embeddings. The modified SAM mask decoder, equipped with IPEF, fuses prompt and image information and outputs the final segmentation (Zhuo et al., 23 Jul 2025). This division of labor is central to the method’s design logic: AGM enables full automation, while IPEF reduces the damage caused by residual prompt errors.

The medical motivation is domain shift. In single-source domain generalization, the system is trained only on one source domain but must operate on unseen target domains with shifts in appearance, contrast, texture, and acquisition conditions (Zhuo et al., 23 Jul 2025). The paper positions AGM as the mechanism that makes SAM usable under this regime without requiring a radiologist or other specialist to specify the target region at inference time.

3. Architecture and auto-prompting mechanism

Architecturally, AGM is implemented as an additional trainable segmentation branch separate from SAM itself, and the paper states that Meganet is used as the backbone network for AGM in the prompt-generation stage (Zhuo et al., 23 Jul 2025). AGM is not a branch inside SAM’s ViT image encoder; it is an auxiliary segmentation network whose output is converted into SAM prompts (Zhuo et al., 23 Jul 2025).

The source and target data are formulated as

$$\mathcal{D}_{S}=\{({x}_{s}^i, {y}_{s}^i) \mid i=1,2,\ldots,{N_s}\}, \qquad \mathcal{D}_{T}=\{{x}_{t}^j \mid j=1,2,\ldots,{N_t}\}.$$

AGM is trained only on the labeled source domain, using source-domain images and source masks, and at inference it runs on target-domain images to generate prompts automatically (Zhuo et al., 23 Jul 2025). The segmentation loss for AGM is denoted $\mathcal{L}_{\text{sup}}$ and is described as a combination of cross-entropy loss and Dice loss (Zhuo et al., 23 Jul 2025).

The prompt-generation procedure is explicit. AGM predicts a preliminary segmentation mask. A Breadth-First Search connected-component procedure then retains only the largest connected component and discards all smaller ones. The bounding rectangle of this cleaned mask is used as the bounding-box prompt for SAM (Zhuo et al., 23 Jul 2025). The paper does not provide an explicit formula for box coordinates, but it does specify the operational sequence: preliminary mask prediction, largest connected component via BFS, bounding box extraction, and prompt encoding by SAM (Zhuo et al., 23 Jul 2025).

This pipeline establishes the article’s defining meaning of “auto-prompted.” AGM does not synthesize natural-language instructions or optimize prompts by search. Instead, it automatically converts an input medical image into a geometrically grounded SAM prompt, specifically a bounding box, through a learned segmentation stage and a deterministic post-processing stage (Zhuo et al., 23 Jul 2025).

4. SUFM: shallow-feature uncertainty modeling

The key technical addition within AGM is SUFM, which is intended to improve prompt quality by improving AGM’s preliminary mask on unseen target domains (Zhuo et al., 23 Jul 2025). The motivation is that domain shift in medical images often appears first in low-level visual characteristics such as color, texture, and contrast, and shallow convolutional layers are especially sensitive to and descriptive of those cues (Zhuo et al., 23 Jul 2025). The paper therefore perturbs feature statistics in the shallow part of the AGM encoder rather than in deep semantic layers.

For an input source image $x_s$, the shallow encoded features are written as

$f \in \mathbb{R}^{B\times C \times H \times W}.$

The paper denotes the channel-wise mean and variance for each instance as $\mu \in \mathbb{R}^{B \times C}$ and $\sigma \in \mathbb{R}^{B \times C}$, and explicitly notes that the notation is somewhat inconsistent typographically, while the intended meaning is standard feature-statistics computation over shallow feature maps (Zhuo et al., 23 Jul 2025). Uncertainty is then defined through the batch variance of these statistics:

$$\begin{aligned} {\Sigma^2_{\mu}({x})} &= {(1/B) \cdot} \sum\nolimits_{i=1}^{B} (\mu({x})-\mathbb{E}_{b}[\mu({x})])^2, \ {\Sigma^2_{\sigma}({x})} &= {(1/B) \cdot} \sum\nolimits_{i=1}^{B} (\sigma({x})-\mathbb{E}_{b}[\sigma({x})])^2. \end{aligned}$$

Here, $\Sigma_{\mu}({x}) \in \mathbb{R}^C$ and $\Sigma_{\sigma}({x}) \in \mathbb{R}^C$ are the uncertainty estimates of the feature mean and feature standard deviation, respectively (Zhuo et al., 23 Jul 2025).

The uncertainty sampling step is

$$\mathcal{D}_{S}=\{({x}_{s}^i, {y}_{s}^i) \mid i=1,2,\ldots,{N_s}\}, \qquad \mathcal{D}_{T}=\{{x}_{t}^j \mid j=1,2,\ldots,{N_t}\}.$$0

These sampled perturbations are then combined with Poisson noise:

$$\mathcal{D}_{S}=\{({x}_{s}^i, {y}_{s}^i) \mid i=1,2,\ldots,{N_s}\}, \qquad \mathcal{D}_{T}=\{{x}_{t}^j \mid j=1,2,\ldots,{N_t}\}.$$1

with

$$\mathcal{D}_{S}=\{({x}_{s}^i, {y}_{s}^i) \mid i=1,2,\ldots,{N_s}\}, \qquad \mathcal{D}_{T}=\{{x}_{t}^j \mid j=1,2,\ldots,{N_t}\}.$$2

Finally, SUFM performs an affine feature transformation:

$$\mathcal{D}_{S}=\{({x}_{s}^i, {y}_{s}^i) \mid i=1,2,\ldots,{N_s}\}, \qquad \mathcal{D}_{T}=\{{x}_{t}^j \mid j=1,2,\ldots,{N_t}\}.$$3

The paper interprets the Gaussian term as modeling continuous distributional uncertainty and the Poisson term as modeling sparse or burst-like perturbations relevant to real medical imaging noise (Zhuo et al., 23 Jul 2025).

In functional terms, SUFM is a feature-statistic perturbation mechanism for shallow representations. Its stated purpose is to diversify shallow features so that AGM is exposed during training to a broader distribution of style-like variations and can later generate more reliable prompts on unseen target domains (Zhuo et al., 23 Jul 2025). A plausible implication is that SUFM should be understood as a domain-generalization device specialized to the prompt-generation stage rather than as a generic segmentation regularizer.

5. Training stages and interaction with SAM

The paper describes two training stages (Zhuo et al., 23 Jul 2025). In the first stage, AGM is trained as a supervised segmentation network on source-domain images only. In the second stage, the SAM segmentation pipeline is trained or fine-tuned with a second loss, $$\mathcal{D}_{S}=\{({x}_{s}^i, {y}_{s}^i) \mid i=1,2,\ldots,{N_s}\}, \qquad \mathcal{D}_{T}=\{{x}_{t}^j \mid j=1,2,\ldots,{N_t}\}.$$4, also described as a combination of cross-entropy loss and Dice loss (Zhuo et al., 23 Jul 2025).

In the SAM stage, the pre-trained ViT-B image encoder and prompt encoder are frozen (Zhuo et al., 23 Jul 2025). This implies that AGM and the downstream SAM segmentation pipeline are optimized in separate supervised stages with separate losses, rather than as a single joint graph with gradients flowing through SAM back into AGM (Zhuo et al., 23 Jul 2025). AGM supplies prompts; SAM consumes them.

IPEF is the component that complements AGM downstream. The paper states that image embeddings are extracted from global attention layers of the SAM ViT-B image encoder at channels 256, 128, and 64. These are concatenated with prompt embeddings along the channel dimension, enhanced using Squeeze-and-Excitation Residual blocks (SEResBlocks), and then progressively upsampled with skip connections to produce high-resolution features for mask decoding (Zhuo et al., 23 Jul 2025). The stated rationale is that SAM’s original transformer-based mask decoder emphasizes global and low-frequency information and lacks progressive upsampling and skip connections, which can make it brittle when prompts are inaccurate (Zhuo et al., 23 Jul 2025).

Implementation details are also reported. In the AGM stage, images are resized to $$\mathcal{D}_{S}=\{({x}_{s}^i, {y}_{s}^i) \mid i=1,2,\ldots,{N_s}\}, \qquad \mathcal{D}_{T}=\{{x}_{t}^j \mid j=1,2,\ldots,{N_t}\}.$$5, batch size is 8, training runs for 200 iterations, the learning rate is 0.001, the optimizer is Adam decay, and experiments run on an NVIDIA RTX 3090 (Zhuo et al., 23 Jul 2025). In the SAM fine-tuning stage, images are resized to $$\mathcal{D}_{S}=\{({x}_{s}^i, {y}_{s}^i) \mid i=1,2,\ldots,{N_s}\}, \qquad \mathcal{D}_{T}=\{{x}_{t}^j \mid j=1,2,\ldots,{N_t}\}.$$6, batch size is 4, training runs for 100 iterations, and the pre-trained ViT-B image and prompt encoders remain frozen (Zhuo et al., 23 Jul 2025).

6. Empirical evidence, interpretation, and limitations

The most direct evidence for AGM comes from the module ablations. With SAM plus AGM but without SUFM or IPEF, the average Dice on prostate MRI is 81.31\%. Adding SUFM to AGM raises this to 83.62\%, an improvement of 2.31 points overall. The per-domain gains are especially large on difficult target sites: site C improves from 75.02\% to 82.62\%, and site E from 69.24\% to 73.43\% (Zhuo et al., 23 Jul 2025). Adding IPEF instead of SUFM gives 83.52\%, while combining both SUFM and IPEF yields the best result, 84.53\% (Zhuo et al., 23 Jul 2025). The paper explicitly interprets this as evidence that SUFM alleviates source-target domain shift and helps AGM avoid poor-quality prompts.

A second ablation examines where SUFM should be inserted. The best performance occurs when SUFM is inserted between Layer 0 and Layer 1, i.e. very early in the convolutional encoder. Table 4 reports a preliminary prediction Dice of 81.51\% and final prediction Dice of 84.53\% for this early insertion, while performance degrades steadily when the module is moved deeper, reaching a final prediction Dice of 77.25\% at position 5 (Zhuo et al., 23 Jul 2025). This directly supports the paper’s claim that uncertainty should be modeled in shallow features.

A third ablation studies the perturbation distribution used in SUFM. For preliminary prediction, Dice scores are 72.46\% for Gaussian-only, 78.93\% for Poisson-only, and 81.51\% for the combined “United” mechanism. For final prediction, the corresponding values are 76.96\%, 82.53\%, and 84.53\% (Zhuo et al., 23 Jul 2025). This is the clearest quantitative support for combining Gaussian uncertainty estimation with Poisson noise injection.

The paper also reports a robustness comparison with ground-truth prompts. SAM with ground-truth prompts achieves 90.53\% average Dice, and SAM+IPEF with ground-truth prompts improves to 91.54\%, a gain of 1.01 points. FA-SAM, which uses AGM-generated prompts rather than ground truth, reaches 84.53\% (Zhuo et al., 23 Jul 2025). This establishes an important boundary condition: automatically generated prompts remain worse than ideal manual or ground-truth prompts, even though they are good enough to enable fully automated segmentation. A plausible implication is that AGM should be understood as an enabling auto-prompting mechanism rather than as a perfect substitute for oracle prompt quality.

The limitations are largely implicit but clear. AGM is supervised only with source labels, so target-domain prompt generation remains approximate; poor prompts cannot be fully avoided (Zhuo et al., 23 Jul 2025). The prompt extraction mechanism is deliberately simple—preliminary mask, BFS largest-component filtering, and a derived box—and the paper does not report a dedicated prompt-quality metric, does not compare AGM against alternative auto-prompt strategies beyond the AGM-without-SUFM variant, and does not give algorithmic pseudocode or exact box parameterization formulas (Zhuo et al., 23 Jul 2025). The paper also does not provide AGM parameter count, FLOPs, or explicit runtime overhead, although AGM clearly adds an extra segmentation network before SAM and therefore increases computation relative to vanilla prompted SAM (Zhuo et al., 23 Jul 2025).

In summary, Auto-prompted Generation Model refers to a task-specific prompt-generation branch for fully automated SAM in medical image segmentation under single-source domain generalization (Zhuo et al., 23 Jul 2025). Its defining components are an auxiliary segmentation network, shallow-feature uncertainty modeling through SUFM, and geometric prompt extraction through largest-connected-component filtering and bounding-box conversion. Its significance lies not in introducing a universal prompt-generation paradigm, but in showing that SAM-style prompt dependence can be operationally mitigated by learned auto-prompting on the image side, with measurable gains under domain shift (Zhuo et al., 23 Jul 2025).