---
title: Auto-prompted Generation Model (AGM)
url: https://www.emergentmind.com/topics/auto-prompted-generation-model-agm
type: topic
---

# Auto-prompted Generation Model (AGM)

Auto-prompted Generation Model (AGM) denotes, in the FA-SAM framework, an auxiliary trainable segmentation branch that automatically generates bounding-box prompts for the Segment Anything Model (SAM) in single-source domain generalization for medical image segmentation [2507.17281]. Its purpose is to remove the dependence of SAM on domain-specific expert-annotated prompts, especially bounding boxes, and thereby enable fully automated segmentation at test time in cross-domain settings where only a single labeled source domain is available during training [2507.17281]. Within FA-SAM, AGM operates upstream of SAM: it predicts a preliminary mask from the input medical image, converts that mask into a bounding-box prompt, and supplies the prompt to a frozen SAM prompt encoder, after which a modified SAM decoder produces the final segmentation [2507.17281].

## 1. Terminological scope and acronym ambiguity

The acronym “AGM” is heavily overloaded in the arXiv literature. In classical numerical analysis and computational mathematics, AGM commonly denotes the arithmetic-geometric mean of Gauss and Legendre, which underlies high-precision algorithms for $\pi$ and elementary functions [1802.07558], and also rapid approximations to the period of the simple pendulum [1202.2782]. In machine learning, AGM has also been expanded as “Adaptive Generation Model,” a stacking-like ensemble method for tabular classification [2009.06332]. In logic and belief change, AGM refers to the Alchourrón–Gärdenfors–Makinson framework for belief revision [2112.13557].

In the present sense, AGM means **Auto-prompted Generation Model** and is specific to the FA-SAM framework for medical image segmentation under single-source domain generalization [2507.17281]. The term does not designate a general-purpose prompt optimization architecture across arbitrary LLM tasks; rather, it names a component whose output is a SAM-compatible bounding-box prompt derived automatically from a preliminary segmentation mask [2507.17281].

## 2. Functional position within FA-SAM

FA-SAM is organized around two main innovations: the AGM branch, which includes a Shallow Feature Uncertainty Modeling (SUFM) module, and the Image-Prompt Embedding Fusion (IPEF) module integrated into the SAM mask decoder [2507.17281]. Their roles are complementary. AGM is responsible for prompt generation: it replaces manual prompt annotation by predicting a coarse mask and converting that prediction into a bounding box for SAM. IPEF addresses the residual brittleness of SAM when prompts are imperfect by improving fusion between image embeddings and prompt embeddings inside the decoder [2507.17281].

The resulting workflow is sequential rather than end-to-end in the strong differentiable sense described in many prompt-conditioned systems. An input image is first processed by AGM to obtain a preliminary segmentation mask. That mask is then post-processed into a bounding box. The box is passed to the frozen SAM prompt encoder; in parallel, the SAM image encoder provides image embeddings. The modified SAM mask decoder, equipped with IPEF, fuses prompt and image information and outputs the final segmentation [2507.17281]. This division of labor is central to the method’s design logic: AGM enables full automation, while IPEF reduces the damage caused by residual prompt errors.

The medical motivation is domain shift. In single-source domain generalization, the system is trained only on one source domain but must operate on unseen target domains with shifts in appearance, contrast, texture, and acquisition conditions [2507.17281]. The paper positions AGM as the mechanism that makes SAM usable under this regime without requiring a radiologist or other specialist to specify the target region at inference time.

## 3. Architecture and auto-prompting mechanism

Architecturally, AGM is implemented as an additional trainable segmentation branch separate from SAM itself, and the paper states that **Meganet** is used as the backbone network for AGM in the prompt-generation stage [2507.17281]. AGM is not a branch inside SAM’s ViT image encoder; it is an auxiliary segmentation network whose output is converted into SAM prompts [2507.17281].

The source and target data are formulated as
$$
\mathcal{D}_{S}=\{({x}_{s}^i, {y}_{s}^i) \mid i=1,2,\ldots,{N_s}\},
\qquad
\mathcal{D}_{T}=\{{x}_{t}^j \mid j=1,2,\ldots,{N_t}\}.
$$
AGM is trained only on the labeled source domain, using source-domain images and source masks, and at inference it runs on target-domain images to generate prompts automatically [2507.17281]. The segmentation loss for AGM is denoted $\mathcal{L}_{\text{sup}}$ and is described as a combination of cross-entropy loss and Dice loss [2507.17281].

The prompt-generation procedure is explicit. AGM predicts a preliminary segmentation mask. A Breadth-First Search connected-component procedure then retains only the largest connected component and discards all smaller ones. The bounding rectangle of this cleaned mask is used as the bounding-box prompt for SAM [2507.17281]. The paper does not provide an explicit formula for box coordinates, but it does specify the operational sequence: preliminary mask prediction, largest connected component via BFS, bounding box extraction, and prompt encoding by SAM [2507.17281].

This pipeline establishes the article’s defining meaning of “auto-prompted.” AGM does not synthesize natural-language instructions or optimize prompts by search. Instead, it automatically converts an input medical image into a geometrically grounded SAM prompt, specifically a bounding box, through a learned segmentation stage and a deterministic post-processing stage [2507.17281].

## 4. SUFM: shallow-feature uncertainty modeling

The key technical addition within AGM is SUFM, which is intended to improve prompt quality by improving AGM’s preliminary mask on unseen target domains [2507.17281]. The motivation is that domain shift in medical images often appears first in low-level visual characteristics such as color, texture, and contrast, and shallow convolutional layers are especially sensitive to and descriptive of those cues [2507.17281]. The paper therefore perturbs feature statistics in the shallow part of the AGM encoder rather than in deep semantic layers.

For an input source image $x_s$, the shallow encoded features are written as
$$
f \in \mathbb{R}^{B\times C \times H \times W}.
$$
The paper denotes the channel-wise mean and variance for each instance as $\mu \in \mathbb{R}^{B \times C}$ and $\sigma \in \mathbb{R}^{B \times C}$, and explicitly notes that the notation is somewhat inconsistent typographically, while the intended meaning is standard feature-statistics computation over shallow feature maps [2507.17281]. Uncertainty is then defined through the batch variance of these statistics:
$$
\begin{aligned}
{\Sigma^2_{\mu}({x})} &= {(1/B) \cdot} \sum\nolimits_{i=1}^{B} (\mu({x})-\mathbb{E}_{b}[\mu({x})])^2, \\
{\Sigma^2_{\sigma}({x})} &= {(1/B) \cdot} \sum\nolimits_{i=1}^{B} (\sigma({x})-\mathbb{E}_{b}[\sigma({x})])^2.
\end{aligned}
$$
Here, $\Sigma_{\mu}({x}) \in \mathbb{R}^C$ and $\Sigma_{\sigma}({x}) \in \mathbb{R}^C$ are the uncertainty estimates of the feature mean and feature standard deviation, respectively [2507.17281].

The uncertainty sampling step is
$$
\mu^*_\mu \sim \mathcal{N}(\mu,\Sigma^2_\mu), \qquad
\mu^*_\sigma \sim \mathcal{N}(\sigma,\Sigma^2_\sigma).
$$
These sampled perturbations are then combined with Poisson noise:
$$
\begin{aligned}
\beta({x})  & = \mu({x}) + \mu^*_{\mu} \cdot {\Sigma_{\mu}({x})} + \epsilon_\mu, \\
\gamma({x}) & = \sigma({x}) + \mu^*_{\sigma} \cdot {\Sigma_{\sigma}({x})} + \epsilon_\sigma,
\end{aligned}
$$
with
$$
\epsilon_\mu \sim \mathrm{Poisson}(|\mu^*_\mu|), \qquad
\epsilon_\sigma \sim \mathrm{Poisson}(|\mu^*_\sigma|).
$$
Finally, SUFM performs an affine feature transformation:
$$
f = \gamma({x}) \cdot ({x} - \mu({x})) \cdot {\sigma({x})}^{-1} + \beta({x}).
$$
The paper interprets the Gaussian term as modeling continuous distributional uncertainty and the Poisson term as modeling sparse or burst-like perturbations relevant to real medical imaging noise [2507.17281].

In functional terms, SUFM is a feature-statistic perturbation mechanism for shallow representations. Its stated purpose is to diversify shallow features so that AGM is exposed during training to a broader distribution of style-like variations and can later generate more reliable prompts on unseen target domains [2507.17281]. A plausible implication is that SUFM should be understood as a domain-generalization device specialized to the prompt-generation stage rather than as a generic segmentation regularizer.

## 5. Training stages and interaction with SAM

The paper describes **two training stages** [2507.17281]. In the first stage, AGM is trained as a supervised segmentation network on source-domain images only. In the second stage, the SAM segmentation pipeline is trained or fine-tuned with a second loss, $\mathcal{L}_{\text{seg}}$, also described as a combination of cross-entropy loss and Dice loss [2507.17281].

In the SAM stage, the pre-trained **ViT-B image encoder** and **prompt encoder** are frozen [2507.17281]. This implies that AGM and the downstream SAM segmentation pipeline are optimized in separate supervised stages with separate losses, rather than as a single joint graph with gradients flowing through SAM back into AGM [2507.17281]. AGM supplies prompts; SAM consumes them.

IPEF is the component that complements AGM downstream. The paper states that image embeddings are extracted from global attention layers of the SAM ViT-B image encoder at channels **256, 128, and 64**. These are concatenated with prompt embeddings along the channel dimension, enhanced using **Squeeze-and-Excitation Residual blocks (SEResBlocks)**, and then progressively upsampled with skip connections to produce high-resolution features for mask decoding [2507.17281]. The stated rationale is that SAM’s original transformer-based mask decoder emphasizes global and low-frequency information and lacks progressive upsampling and skip connections, which can make it brittle when prompts are inaccurate [2507.17281].

Implementation details are also reported. In the AGM stage, images are resized to **$512 \times 512$**, batch size is **8**, training runs for **200 iterations**, the learning rate is **0.001**, the optimizer is **Adam decay**, and experiments run on an **NVIDIA RTX 3090** [2507.17281]. In the SAM fine-tuning stage, images are resized to **$1024 \times 1024$**, batch size is **4**, training runs for **100 iterations**, and the pre-trained ViT-B image and prompt encoders remain frozen [2507.17281].

## 6. Empirical evidence, interpretation, and limitations

The most direct evidence for AGM comes from the module ablations. With SAM plus AGM but without SUFM or IPEF, the average Dice on prostate MRI is **81.31\%**. Adding SUFM to AGM raises this to **83.62\%**, an improvement of **2.31** points overall. The per-domain gains are especially large on difficult target sites: site C improves from **75.02\%** to **82.62\%**, and site E from **69.24\%** to **73.43\%** [2507.17281]. Adding IPEF instead of SUFM gives **83.52\%**, while combining both SUFM and IPEF yields the best result, **84.53\%** [2507.17281]. The paper explicitly interprets this as evidence that SUFM alleviates source-target domain shift and helps AGM avoid poor-quality prompts.

A second ablation examines where SUFM should be inserted. The best performance occurs when SUFM is inserted between **Layer 0 and Layer 1**, i.e. very early in the convolutional encoder. Table 4 reports a preliminary prediction Dice of **81.51\%** and final prediction Dice of **84.53\%** for this early insertion, while performance degrades steadily when the module is moved deeper, reaching a final prediction Dice of **77.25\%** at position **5** [2507.17281]. This directly supports the paper’s claim that uncertainty should be modeled in shallow features.

A third ablation studies the perturbation distribution used in SUFM. For preliminary prediction, Dice scores are **72.46\%** for Gaussian-only, **78.93\%** for Poisson-only, and **81.51\%** for the combined “United” mechanism. For final prediction, the corresponding values are **76.96\%**, **82.53\%**, and **84.53\%** [2507.17281]. This is the clearest quantitative support for combining Gaussian uncertainty estimation with Poisson noise injection.

The paper also reports a robustness comparison with ground-truth prompts. SAM with ground-truth prompts achieves **90.53\%** average Dice, and SAM+IPEF with ground-truth prompts improves to **91.54\%**, a gain of **1.01** points. FA-SAM, which uses AGM-generated prompts rather than ground truth, reaches **84.53\%** [2507.17281]. This establishes an important boundary condition: automatically generated prompts remain worse than ideal manual or ground-truth prompts, even though they are good enough to enable fully automated segmentation. A plausible implication is that AGM should be understood as an enabling auto-prompting mechanism rather than as a perfect substitute for oracle prompt quality.

The limitations are largely implicit but clear. AGM is supervised only with source labels, so target-domain prompt generation remains approximate; poor prompts cannot be fully avoided [2507.17281]. The prompt extraction mechanism is deliberately simple—preliminary mask, BFS largest-component filtering, and a derived box—and the paper does not report a dedicated prompt-quality metric, does not compare AGM against alternative auto-prompt strategies beyond the AGM-without-SUFM variant, and does not give algorithmic pseudocode or exact box parameterization formulas [2507.17281]. The paper also does not provide AGM parameter count, FLOPs, or explicit runtime overhead, although AGM clearly adds an extra segmentation network before SAM and therefore increases computation relative to vanilla prompted SAM [2507.17281].

In summary, Auto-prompted Generation Model refers to a task-specific prompt-generation branch for fully automated SAM in medical image segmentation under single-source domain generalization [2507.17281]. Its defining components are an auxiliary segmentation network, shallow-feature uncertainty modeling through SUFM, and geometric prompt extraction through largest-connected-component filtering and bounding-box conversion. Its significance lies not in introducing a universal prompt-generation paradigm, but in showing that SAM-style prompt dependence can be operationally mitigated by learned auto-prompting on the image side, with measurable gains under domain shift [2507.17281].

Source: https://www.emergentmind.com/topics/auto-prompted-generation-model-agm