System-Prompt Grounding (P-ground)

• System-Prompt Grounding (P-ground) is a metric defining how slight linguistic variations in prompts lead to measurable output instability in vision–language systems.
• The methodology employs controlled pipelines using cosine similarity, PCA, and argmax selection to quantify discrepancies in 2D and 3D object detection and anomaly detection tasks.
• Key insights highlight that current prompt ensembling and discrete selection techniques undermine semantic accuracy, urging the development of more robust, prompt-invariant architectures.

System-Prompt Grounding (P-ground) denotes model behavior and methodology at the intersection of prompt engineering, object detection, and language–vision alignment, with a focus on how the precise formulation of prompts determines grounding outputs. Widely studied in both 2D and 3D settings, P-ground addresses the technical and algorithmic consequences when variations in prompt content or class labels modify the regions or objects grounded by a model, even when those prompts are semantically equivalent. The phenomenon exposes nontrivial instability and brittleness in vision–language systems, with significant implications for open-vocabulary detection, anomaly detection, and multi-modal captioning (Deka et al., 18 Apr 2026, Cheung et al., 2024, Luo et al., 2024).

1. Phenomenon of Prompt Sensitivity and P-ground Definition

In vision–language grounding, prompt sensitivity refers to the observation that semantically overlapping queries such as “a person,” “a human,” and “a pedestrian” can elicit different object selections or bounding boxes from a detector, despite encoding the same real-world referent (Deka et al., 18 Apr 2026). System-Prompt Grounding (P-ground) formalizes this phenomenon, focusing on the extent to which the output of grounding models remains invariant—or fails to remain invariant—across minor linguistic reformulations.

The core assumption of many open-vocabulary detection pipelines is that the model will be robust to such prompt variability. P-ground is the metricization of the deviation from this assumption, quantifying argmax-induced brittleness in similarity-based selection schemes.

2. Methodological Pipelines in P-ground Studies

P-ground has been operationalized and scrutinized through tightly controlled experimental pipelines that decouple proposal generation from language-conditioned selection. Two dominant paradigms have emerged:

1. 2D Vision–Language Grounding Pipeline (Deka et al., 18 Apr 2026):
   • Object proposals are generated using a model such as DETR; bounding boxes for a specified class (“person”) are selected above a confidence threshold.
   • Each boxed region is embedded using CLIP’s image encoder.
   • Multiple semantically similar prompts are encoded via CLIP’s text encoder.
   • Cosine similarity is computed between every box and every prompt; an argmax over similarity scores per prompt determines the selected grounding.
2. Automated Industrial Anomaly Detection Pipeline (Cheung et al., 2024):
   • GPT-3 automatically generates descriptive “system prompts” for both normal and anomalous states of a given product class.
   • Objects are localized in images using Grounding DINO conditioned on class prompts.
   • Cropped image regions are embedded using CLIP; class and anomaly prompts are similarly embedded.
   • An anomaly score is derived by taking dot products between image and prompt embeddings, classifying the image based on relative scores.

The P-ground methodology in both settings involves comparing the sets of regions selected under various close prompts to directly measure prompt-induced output instability.

3. Mathematical Formulation and Instability Metrics

The formalism underlying P-ground employs standard representation learning constructs but with specific metrics to quantify prompt-driven variability. For image $i$ and set of $K$ prompts, the prompt instability is

$$\text{Instability}_i = \bigl| \left\{ b^*(p_k) \,\big|\, k=1,\dots, K \right\} \bigr|,$$

where $b^*(p_k)$ is the selected bounding box for prompt $p_k$. The mean instability is averaged over all test images:

$$\text{Mean Instability} = \frac{1}{N}\sum_{i=1}^{N} \text{Instability}_i.$$

(Deka et al., 18 Apr 2026) reports a mean instability of 2.11 across six human-related prompts on 263 COCO images, with 55% of images exhibiting different boxes per prompt. Cosine similarity is used for selection:

$$s(b, p) = \frac { f_{\text{img}}(b)^\top f_{\text{text}}(p) } { \|f_{\text{img}}(b)\| \cdot \|f_{\text{text}}(p)\| },$$

where $f_{\text{img}}$ and $f_{\text{text}}$ are the CLIP image and text encoders, respectively.

In anomaly detection pipelines, the final anomaly score is given by

$$\text{score}(I) = \frac { s_{\text{anom}} } { s_{\text{anom}} + s_{\text{norm}} }$$

where $K$0 and $K$1 (Cheung et al., 2024).

4. Structural Analysis of Prompt-Induced Variability

Principal Component Analysis (PCA) applied to prompt-conditioned score vectors reveals that variability is not isotropic noise but rather lies along interpretable, structured directions—distinct prompts map to distinct “lobes” in score space (Deka et al., 18 Apr 2026). For each bounding box, the vector of CLIP scores across prompts forms a point in $K$2 (e.g., a 6D vector for six prompts):

$K$3

Agglomerating these for all boxes and applying PCA demonstrates that prompt differences produce systematic, directional effects in grounding outputs, often tracing to subtle semantic or visual biases (e.g., postural emphasis, accessory detection, or scene context).

5. Prompt Ensembling and the Limits of Robustness

A prevalent remedy—averaging similarity scores across multiple overlapping prompts (“prompt ensembling”)—does not eliminate P-ground instability. Instead, ensembling suppresses semantically salient, high-variance selections in favor of visually generic, mid-score regions (e.g., partial objects or background patches). This results in reduced semantic correctness, despite increased stability, and provides no accuracy gains; in numerous cases, the ensemble output is qualitatively inferior (Deka et al., 18 Apr 2026). The propensity for argmax to produce abrupt, non-Lipschitz changes in outputs under small prompt shifts further exposes the inadequacy of ensembling for prompt invariance.

6. Text Embedding Proximity Versus Grounding Disagreement

Cosine proximities between prompt embeddings in text space only partially explain grounding disagreement. Correlation analysis yields $K$4 (with $K$5), indicating that only 34% of the variance in grounding disagreement is accounted for by distance in CLIP text space (Deka et al., 18 Apr 2026). The remaining 66% is attributed to the argmax mechanism’s sensitivity: small perturbations in scores across highly similar prompts can flip the maximal score selection, thereby producing unpredictable spatial shifts even for nearly co-located embeddings.

7. Extensions, Applications, and Limitations

Beyond 2D visual grounding, P-ground methodology has been extended to multi-modal industrial anomaly detection and 3D vision-language tasks:

• Automated Prompt Generation and Anomaly Detection: P-ground in (Cheung et al., 2024) unifies automatic LLM-based prompt generation, text-conditioned object localization, and zero-shot image-text matching, achieving strong AUROC/AUPR in MVTec-AD and VisA product inspection, with explicit separation between normal and anomalous system prompts.
• Prompt-Based 3D Vision Applications: In 3DGCTR (Luo et al., 2024), prompt-based localization is exploited for dense captioning and 3D visual grounding, demonstrating that joint prompt-conditioned multi-task training yields state-of-the-art results on ScanRefer.

Limitations include reliance on prompt engineering quality, fixed thresholds that may require per-class or per-dataset calibration, and the lack of pixel-level grounding in some applications. Scalability to out-of-vocabulary classes and very long or complex prompts remains a challenge, and the strong influence of discrete argmax selection poses fundamental obstacles to robustness.

Table: Instability and Prompt Sensitivity Metrics Across Tasks

Study/Task	Instability Metric (Mean)	Coverage / Impact
(Deka et al., 18 Apr 2026) Vision-Language Grounding	2.11 distinct boxes/image	55% of images with disagreement
(Cheung et al., 2024) Anomaly Detection	AUROC/AUPR: 0.932/0.966 (MVTec-AD)	System prompt-based anomaly scoring

The table highlights both the average degree of instability under prompt rephrasings as well as downstream effectiveness when automatic prompt generation is incorporated.

---

System-Prompt Grounding exposes fundamental brittleness in vision–language grounding models, rooted in argmax selection and prompt embedding geometry, and underscores the need for architectures and methodologies that move beyond raw cosine similarity and discrete selection to achieve robust, prompt-invariant behavior in visual grounding and multimodal understanding (Deka et al., 18 Apr 2026, Cheung et al., 2024, Luo et al., 2024).