Prompt-Guided Semantic Cropping

• Prompt-guided Semantic-aware Cropping (PSC) is a set of methods that use textual or visual prompts to select image regions based on semantic relevance rather than mere visual saliency.
• Techniques such as entity-conditioned crop ranking, prompt-conditioned detection with OWL-ViT, and feature-level token cropping in videos highlight its diverse applications.
• PSC improves tasks like zero-shot classification and video-language inference by refining region selection to balance object inclusion, aesthetics, and computational efficiency.

Prompt-guided Semantic-aware Cropping (PSC) denotes a family of methods in which a prompt—such as a class name, a natural-language description, a question, or an exemplar image—guides the selection or refinement of an image region so that the retained crop is semantically aligned with a target concept rather than merely visually salient. In the literature, this idea appears in several task-specific forms: entity-conditioned crop ranking that fuses aesthetic and semantic maps in semantic image cropping (Corcoll, 2021), prompt-conditioned aesthetic cropping built on OWL-ViT in ClipCrop (Zhong et al., 2022), guided object-centric cropping for zero-shot CLIP classification in GC-CLIP (Saranrittichai et al., 2023), and prompt-guided spatial RoI cropping for token-efficient video-language inference in Free Video-LLM (Han et al., 2024). Across these formulations, the central claim is consistent: cropping becomes semantically meaningful only when region selection is conditioned on the target entity or task prompt.

1. Definition and conceptual boundaries

Semantic image cropping was formalized as the problem: given an image $I$, an aspect ratio $r$, and an entity $e$, provide a crop $C$ with aspect ratio $r$ that preserves the high-aesthetic and good-composition areas of $I$ including $e$ (Corcoll, 2021). This definition distinguishes semantic cropping from aesthetic-only cropping, whose objective is to maximize visual appeal without explicitly preserving a specified subject. The distinction matters when multiple subjects are spatially separated or when a fixed aspect ratio forces a trade-off between entity inclusion and composition.

In zero-shot classification, the same principle is reframed around recognition rather than presentation. CLIP’s image encoder is designed to summarize the entire image, and this genericity can be harmful for closed-set classification when large background regions induce high similarity to distractor texts. The GC-CLIP formulation addresses this by using class prompts to guide an open-vocabulary detector toward semantically relevant regions and cropping the image around the resulting box so that CLIP focuses on the object of interest (Saranrittichai et al., 2023).

In conditioned aesthetic cropping, ClipCrop operationalizes prompt guidance with either text or image queries. The method uses OWL-ViT to align the prompt with image tokens and then refines the prompt-aligned region into an aesthetic-aware crop using a transformer decoder (Zhong et al., 2022). In video-language inference, the same semantic principle is applied in feature space rather than pixel space: Free Video-LLM uses the task prompt to score patch tokens, selects the most relevant spatial locations, and crops a single region of interest per frame to reduce token count while preserving prompt-relevant evidence (Han et al., 2024).

Work	Prompt form	Cropping mechanism
"Semantic Image Cropping" (Corcoll, 2021)	Entity name	Fusion of aesthetic and semantic maps over candidate crops
"ClipCrop: Conditioned Cropping Driven by Vision-LLM" (Zhong et al., 2022)	Text or image query	OWL-ViT prompt grounding plus decoder-based crop refinement
"Zero-Shot Visual Classification with Guided Cropping" (Saranrittichai et al., 2023)	Class names or phrases	Prompt-guided OWL-ViT box selection plus object-centric CLIP scoring
"Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs" (Han et al., 2024)	Raw task prompt	Prompt-conditioned RoI cropping over feature maps

A common misconception is to equate PSC with generic saliency cropping. The semantic-aware criterion is stricter: the selected region is hypothesized to correspond to a named concept or prompt-relevant content, whereas generic saliency or attention mechanisms may prefer visually dominant but task-irrelevant regions such as water, sky, or ground.

2. GC-CLIP as zero-shot PSC

GC-CLIP provides a direct instantiation of PSC for zero-shot visual classification. The input consists of an image $x$ and a closed set of candidate classes $C=\{c\}$, each associated with one or more classification prompts $t_c$. The pipeline first runs conventional zero-shot CLIP on the full image, obtains preliminary logits over all classes, and restricts refinement to the top-$r$0 classes $r$1 with $r$2 (Saranrittichai et al., 2023).

This top-$r$3 restriction serves two explicit roles. First, the correct class is highly likely to be in the shortlist: on ImageNetS919, top-5 accuracy is $r$4 versus $r$5 top-1, and on CUB it is $r$6 versus $r$7. Second, visually similar top classes tend to yield similar detector boxes, making shared crop refinement effective.

Prompt-guided detection is then performed with OWL-ViT using the class names $r$8 for each $r$9. The robust strategy is multi-pass detection: OWL-ViT is run separately for each prompt and the candidate sets are unioned,

$e$0

The primary box $e$1 is selected as the candidate with the highest detection confidence score. Boxes are converted to square boxes and clipped to image boundaries before cropping.

Crop generation can be performed without augmentation, with Multi-Margin PSC (MAug), or with Random Crop augmentation (RAug). In single-margin PSC, the selected box is enlarged by a margin $e$2:

$e$3

with $e$4 and default $e$5. MAug generates $e$6 crops centered on $e$7 with $e$8, thus ranging from a tight object crop to the full image. RAug instead creates $e$9 random square crops with widths in $C$0 and $C$1.

The cropped views are resized to the CLIP input resolution, and CLIP is run only on the shortlisted classes $C$2. If multiple descriptive templates are available for a class, per-class logits are averaged across templates. Logits are then averaged across crops, and the final prediction is the $C$3 over $C$4.

The empirical gains are concentrated in small-object regimes. With ViT-B/32 and descriptions on ImageNetS919-SM, the CLIP baseline is $C$5, GC-CLIP without augmentation reaches $C$6, and GC-CLIP + MAug reaches $C$7, a gain of $C$8 points over baseline. On CUB-SM with the same backbone and prompt type, the baseline is $C$9, GC-CLIP without augmentation reaches $r$0, and GC-CLIP + RAug reaches $r$1. With ViT-L/14 and descriptions on ImageNetS919-SM, the baseline is $r$2 and GC-CLIP + MAug reaches $r$3. Improvements on full datasets are smaller but remain positive with augmentation, for example from $r$4 to $r$5 on ImageNetS919 for ViT-B/32 with category prompts.

Ablations clarify the role of semantics in the crop. Guided cropping helps when $r$6, but overly tight crops with $r$7 to $r$8 can hurt accuracy by obscuring object boundaries. As the maximum object size decreases from $r$9 down to $I$0, the gap between CLIP and GC-CLIP widens, and MAug outperforms RAug in small-object regimes. OWL-ViT is useful as a box extractor but weak as a direct classifier on ImageNetS919, where it achieves only $I$1 top-1 and $I$2 top-10, indicating that localization quality and fine-grained recognition quality are separable.

3. Objectives, scoring functions, and crop aggregation

The mathematical structure of PSC varies by task, but several recurring terms appear across formulations: a semantic alignment term, an optional aesthetic term, and an aggregation or ranking rule over candidate regions.

In GC-CLIP, standard CLIP zero-shot scoring for a crop $I$3 and class prompt $I$4 is written as

$I$5

while the implementation computes logits as $I$6 (Saranrittichai et al., 2023). With multiple text templates $I$7, the paper averages per-class logits:

$I$8

For $I$9 crops, aggregation is

$e$0

and prediction is $e$1.

In semantic image cropping, the core objects are a CAM-derived aesthetic map $e$2 and a semantic relevance map $e$3. The aesthetic map is

$e$4

and the fused map is

$e$5

Candidate crops $e$6 are ranked by

$e$7

with $e$8 (Corcoll, 2021). This formulation makes the semantics–aesthetics trade-off explicit through the nonnegative weights $e$9 and $x$0.

The same source gives a generalized PSC objective:

$x$1

where $x$2 measures spatial coverage of prompt-relevant regions, $x$3 may be implemented as $x$4, and $x$5 penalizes undesirable crops such as cuts through relevant instances or excessively small windows. If $x$6 and $x$7, the objective reduces to the semantic-cropping fusion score.

In Free Video-LLM, PSC is implemented directly on feature maps. The prompt embedding is

$x$8

For a selected frame $x$9 with feature map $C=\{c\}$0, token-level prompt similarity is

$C=\{c\}$1

Let $C=\{c\}$2. The top-$C=\{c\}$3 locations define a center

$C=\{c\}$4

and the cropped region has size

$C=\{c\}$5

The crop is then flattened to $C=\{c\}$6 tokens (Han et al., 2024).

These formulations suggest that PSC is best understood not as a single algorithm but as a design pattern: prompt-conditioned region selection followed by task-specific scoring, aggregation, or refinement.

4. Aesthetics, entity relevance, and conditioned cropping

The aesthetics–semantics tension is explicit in the earliest semantic cropping formulation. The Semantic Cropping Dataset contains 102 images, each with 2 to 3 entities sufficiently far apart to force a selection under 1:1 crops, and provides 830 individual crops collected from 3 MTurk workers plus the author for each image-entity pair (Corcoll, 2021). Evaluation uses IoU between predicted and ground-truth crops. On this entity-conditioned benchmark, the best weighting depends on what the ground truth values. For the author’s crops, which emphasize aesthetics more while still including the entity, the combined model with $C=\{c\}$7 reaches $C=\{c\}$8 IoU, exceeding both aesthetic-only at $C=\{c\}$9 and semantic-only at $t_c$0. For the MTurk crops, which are tighter and emphasize entity inclusion, semantic-only reaches $t_c$1, above combined at $t_c$2 and aesthetic-only at $t_c$3. On aesthetics-only datasets such as FLMS and Flickr Cropping, semantic-only underperforms, while combined scoring is roughly neutral when aesthetics dominates. This establishes that semantic awareness is not uniformly beneficial; its value depends on whether relevance to a target entity is part of the task.

ClipCrop extends conditioned cropping into an open-vocabulary, prompt-driven regime. The method takes an image $t_c$4 and a user prompt $t_c$5, where the prompt can be either text or an image exemplar. OWL-ViT supplies image tokens $t_c$6, per-object classification embeddings $t_c$7, and initial boxes $t_c$8. For text prompts, the full sentence is parsed with spaCy into $t_c$9 keywords, and both the full text and keywords are encoded with the CLIP text tower. Each text embedding is matched to the most similar classification embedding, producing prompt-aligned image tokens and initial boxes whose union $r$00 forms a semantic proposal region (Zhong et al., 2022).

A DETR-like transformer decoder then refines this union box into $r$01 aesthetic-aware candidates. The decoder is conditioned on the average of the prompt-aligned image tokens added to $r$02 learnable queries, uses all image tokens as keys and values, and predicts offsets $r$03 and scores $r$04. Final boxes are $r$05, and inference returns the top-1 crop by score. The decoder has 6 layers, model dimension 512, 8 attention heads, MLP hidden dimension 2048, and $r$06 learnable queries. Training uses GAIC, one human text description per image, Hungarian assignment, Smooth L1 supervision on normalized aesthetic scores, and mosaic composition to force the model to rely on prompt grounding.

The reported results show the advantage of coupling semantic localization with aesthetic crop refinement. On Horanyi-PR, ClipCrop reaches IoU-Mean/IoU-Max of $r$07, exceeding GAIC+TransVG at $r$08 and OWL-ViT grounding at $r$09. On TextCrop, a benchmark designed for ambiguity in crowded scenes, ClipCrop reaches $r$10, above GAIC+TransVG at $r$11 and the best non-conditioned cropping baseline, Jia et al. (2022), at $r$12. An ablation on Horanyi-PR shows that full text, keyword extraction, and mosaic training all matter: ClipCrop-base without text reaches $r$13, full-text only reaches $r$14, keyword-only reaches $r$15, and the full model reaches $r$16.

Taken together, these results indicate that PSC becomes especially important when ambiguity is structural rather than incidental: multiple plausible targets may coexist, and the prompt determines which one should anchor the crop.

5. Spatial RoI cropping in video-LLMs

In Free Video-LLM, PSC is the spatial half of a broader prompt-guided visual perception framework. The method decouples temporal sampling and spatial cropping: prompt-guided temporal sampling chooses relevant frames, and prompt-guided spatial RoI cropping chooses a single prompt-aligned region within each selected frame (Han et al., 2024).

The system uses CLIP ViT-L/14 at 336-pixel resolution and LLaVA v1.6 as the language backbone. Each frame produces a $r$17 grid of patch tokens, so a full frame contributes 576 visual tokens. The raw prompt string is encoded with the paired CLIP text encoder to obtain $r$18. Frame-level relevance is estimated by global average pooling each frame’s patch tokens into $r$19 and scoring $r$20. The top-$r$21 frames are selected, optionally mixed with a small number of uniformly sampled frames.

Within each selected frame, PSC computes the token-level cosine similarity map $r$22 between the local patch token and the prompt embedding. No external detector or segmenter is used; every patch location is an implicit candidate micro-region. The top-$r$23 scoring locations, with $r$24, define a center, and a fixed-size RoI with area ratio $r$25 is cropped around that center. The crop is taken in feature space, flattened, projected into the LLM embedding space, and concatenated across frames in temporal order. Because cropping occurs after CLIP encoding, no special re-indexing of CLIP’s two-dimensional positional embeddings is required.

The method is explicitly motivated by token efficiency. If a baseline uses $r$26 frames and $r$27 tokens per frame, total visual tokens are $r$28. With temporal sampling and PSC, the retained token count becomes

$r$29

Since transformer self-attention scales as $r$30 with sequence length $r$31, this reduction directly lowers inference cost.

The ablations on MSVD-QA quantify the trade-off. With uniform temporal sampling over 3 frames and no RoI cropping, performance is 71.7 with 864 visual tokens. Prompt-guided temporal sampling over 3 frames and no RoI cropping reaches 75.0 with the same 864 tokens. Prompt-guided temporal sampling plus AvgPool downsampling to 513 tokens reaches 73.8, whereas prompt-guided temporal sampling plus PSC with $r$32 reaches 74.9 with the same 513 tokens. A sweep over $r$33 shows 74.1 at $r$34 with 360 tokens, 74.2 at $r$35 with 408 tokens, 74.9 at $r$36 with 513 tokens, 74.8 at $r$37 with 600 tokens, and 75.0 at $r$38 with 864 tokens. The reported interpretation is that accuracy stabilizes near $r$39 to $r$40, with $r$41 giving a strong accuracy-token trade-off.

At the system level, Free Video-LLM with 1026 tokens achieves 76.8/4.0 on MSVD-QA, 62.9/3.5 on MSRVTT-QA, 53.9/3.4 on ANet-QA, and 75.6/4.0 on TGIF-QA, with average 67.3/3.7. With 2648 tokens, it reaches average 69.1/3.8. On a V100, the 2648-token configuration reports 0.578 s prefill and 20.4 TPS, compared with 0.894 s and 18.2 TPS for IG-VLM at 3456 tokens and 0.961 s and 17.9 TPS for SF-LLaVA at 3680 tokens.

This variant broadens the notion of PSC. In static image classification and conditioned cropping, the crop is usually applied to pixels or bounding boxes. In Free Video-LLM, the crop is a prompt-conditioned truncation of the encoded feature map itself, and its purpose is computational as much as semantic.

6. Failure modes, misconceptions, and research directions

Several failure modes recur across PSC formulations. In GC-CLIP, a missing or incorrect detection among the top-$r$42 prompts can lead to cropping the wrong region; overly small $r$43 can truncate object boundaries; some classes rely on large-scale context; a single primary box is suboptimal in multi-object scenes; and detector inference plus multiple CLIP passes increases latency and cost. The same work also reports that pretrained supervised models may show smaller or negative gains under tight cropping, consistent with their reliance on background shortcuts (Saranrittichai et al., 2023).

In semantic image cropping, failure can originate earlier in the pipeline. WordNet-based mapping from an arbitrary entity string to a detector class can be ambiguous or wrong, detector coverage is limited by the closed COCO label space, and the largest-instance heuristic can select the wrong object when multiple instances are present (Corcoll, 2021). The MTurk-versus-author discrepancy in the semantic cropping dataset also shows that there is no single correct crop when relevance and aesthetics are both valid criteria but weighted differently.

In Free Video-LLM, a single RoI per frame may omit global scene context needed for counting or spatial reasoning; vague or misleading prompts can redirect the crop toward irrelevant patches; and small, occluded, or fast-moving targets may have weak token-level similarity (Han et al., 2024). The proposed mitigations include increasing $r$44, mixing prompt-guided with uniformly sampled frames, or using multiple temporally adjacent frames.

A related misconception is that PSC is necessarily detector-based. The surveyed literature shows at least three distinct mechanisms: explicit box proposals from OWL-ViT in GC-CLIP, map fusion over enumerated crops in semantic image cropping, and prompt-conditioned patch-token saliency in Free Video-LLM. Another misconception is that PSC always improves performance. The reported results are more conditional: semantic signals help when the task genuinely depends on entity relevance, small-object localization, or prompt disambiguation, but they can be neutral or harmful when aesthetics alone is the objective or when critical context lies outside the cropped region.

The proposed research directions follow directly from these limitations. For classification-oriented PSC, suggested extensions include replacing boxes with open-vocabulary segmentation, prompt optimization, joint scoring of detector confidence and CLIP crop similarity, multi-scale crops with dynamic context weighting, region-text alignment refinement, and efficiency-oriented detector or batching improvements (Saranrittichai et al., 2023). For semantic cropping, further work includes detectors with larger label spaces, better entity resolution, saliency-aware fusion, and learned or nonlinear weighting between aesthetics and semantics (Corcoll, 2021). For video PSC, natural extensions are multiple RoIs per frame, overlap control, and hybrid schemes that preserve more global context (Han et al., 2024).

The broader significance of PSC is therefore methodological rather than tied to a single architecture. It provides a way to convert prompts into spatial selectivity, and to do so with objectives that can privilege recognition accuracy, user intent, aesthetics, or inference efficiency depending on the downstream task.