Image Aesthetic Saliency Module (IASM)

• IASM is a design principle that integrates saliency with composition awareness to identify image regions key to aesthetic judgment.
• It is applied in multiple tasks such as cropping, captioning, and composition assessment by leveraging tailored pooling and gradient-based extraction.
• Empirical results show that aesthetics-oriented saliency improves performance, interpretability, and alignment with photographic quality.

Image Aesthetic Saliency Module (IASM) denotes a class of model components that extract or encode image regions most consequential for aesthetic judgment, rather than merely regions that are visually attention-grabbing. In the cited literature, the term is used for a composition- and saliency-aware aesthetic score tensor in image cropping, for a LayerCAM-based aesthetic saliency extractor in aesthetic image captioning, and for a saliency-augmented multi-pattern pooling mechanism that can function as an IASM in composition assessment (Tu et al., 2019, Tao et al., 4 Sep 2025, Zhang et al., 2021). Across these formulations, the common role is to couple saliency with aesthetics-oriented spatial structure so that downstream models prioritize placement, layout, and photographic quality cues.

1. Conceptual scope and definitional variants

Aesthetic saliency is not identical to generic visual saliency. In the aesthetic image captioning formulation, aesthetic saliency denotes image regions that most contribute to perceived beauty and photographic quality, including composition, color harmony, lighting, focus, depth of field, reflections, sky, and foliage; generic visual saliency, by contrast, primarily highlights foreground objects and attention-grabbing subject regions (Tao et al., 4 Sep 2025). In the cropping formulation, saliency-awareness is operationalized by encouraging visually salient regions to have more sensitive aesthetic scores across relative positions in candidate crops (Tu et al., 2019). In the composition-assessment formulation, saliency is injected as partitioned saliency vectors or saliency-weighted pooling so that location and scale cues of salient objects are retained under multiple composition templates (Zhang et al., 2021).

Formulation	Downstream task	Characteristic output
ASM-Net core component	Aesthetic image cropping	Full-resolution position-sensitive aesthetic score tensor $\mathbf{M} \in \mathbb{R}^{H \times W \times L}$
ASE-MLLM IASM	Aesthetic Image Captioning	Scalar saliency map $M_c \in \mathbb{R}^{h \times w}$ and aesthetic saliency features
SAMP as IASM	Image composition assessment	Composition-aware feature $\mathbf{f}_{\text{samp}}$ and pattern weights $w \in \mathbb{R}^P$

This suggests that IASM is best understood not as a single canonical architecture but as a recurring design principle: aesthetic prediction is improved when saliency is made sensitive to composition, partition structure, or aesthetics-oriented token fusion.

2. Position-sensitive IASM in aesthetic image cropping

In ASM-Net, the core IASM-like component builds a single shared, full-resolution, position-sensitive aesthetic score tensor for an image and uses crop-specific composition-aware pooling to score candidate crops (Tu et al., 2019). Let the input image be $I$ with spatial size $H \times W$, and let the backbone be a fully convolutional network derived from VGG16. Multi-scale feature maps from different convolutional layers are upsampled by bilinear interpolation to the input resolution and concatenated along the channel dimension to form a high-resolution multi-scale feature map $F \in \mathbb{R}^{H \times W \times C}$. A $1 \times 1$ convolution then produces the aesthetic score tensor

$\mathbf{M} \in \mathbb{R}^{H \times W \times L},$

where $L$ is the number of composition partitions under a chosen pattern such as a $M_c \in \mathbb{R}^{h \times w}$0 grid, multi-rectangle, multi-direction, or cross-rectangle. Larger scores indicate higher aesthetic quality conditioned on the partition channel.

Composition-awareness is defined through a crop-specific partition mapping. For any axis-aligned crop window $M_c \in \mathbb{R}^{h \times w}$1 and any pixel location $M_c \in \mathbb{R}^{h \times w}$2, the mapping

$M_c \in \mathbb{R}^{h \times w}$3

encodes the partition index of $M_c \in \mathbb{R}^{h \times w}$4 within $M_c \in \mathbb{R}^{h \times w}$5, hence its relative position inside that crop. The composition-aware aesthetic score for a pixel is

$M_c \in \mathbb{R}^{h \times w}$6

and the crop-level aesthetic score is

$M_c \in \mathbb{R}^{h \times w}$7

Because $M_c \in \mathbb{R}^{h \times w}$8 depends on the crop, the same image region can receive different aesthetic scores across different crops. This is the mechanism by which the module models relative placement rather than absolute image coordinates.

For grid patterns with $M_c \in \mathbb{R}^{h \times w}$9, efficient composition-aware pooling is implemented by spatial average pooling within each grid cell of the crop, followed by reshaping to a square matrix $\mathbf{f}_{\text{samp}}$0 whose diagonal aligns cell indices with partition channels. The crop score is then

$\mathbf{f}_{\text{samp}}$1

For non-grid patterns, the paper approximates the pattern by subdividing it into $\mathbf{f}_{\text{samp}}$2 fine cells and copying values from $\mathbf{f}_{\text{samp}}$3 according to which fine cell belongs to which original partition; the same grid pooling is then applied.

Saliency-awareness is imposed during training. An explicit saliency map $\mathbf{f}_{\text{samp}}$4 is computed using the spectral residual method of Hou and Zhang (2007). Two statistics over the $\mathbf{f}_{\text{samp}}$5 channels of $\mathbf{f}_{\text{samp}}$6 are defined:

$\mathbf{f}_{\text{samp}}$7

$\mathbf{f}_{\text{samp}}$8

High $\mathbf{f}_{\text{samp}}$9 identifies content-important regions; high $w \in \mathbb{R}^P$0 identifies composition-sensitive regions. The saliency-aware loss is

$w \in \mathbb{R}^P$1

$w \in \mathbb{R}^P$2

This penalizes large composition sensitivity in non-salient regions and thereby forces composition sensitivity toward salient regions. Ranking supervision is supplied by densely annotated crops and a pairwise hinge loss,

$w \in \mathbb{R}^P$3

with total objective

$w \in \mathbb{R}^P$4

At inference, $w \in \mathbb{R}^P$5 is computed once per image, candidate crops are generated by a two-stage search over predefined anchor boxes and local refinements, and each crop is scored by composition-aware pooling. The FCN pass is $w \in \mathbb{R}^P$6 once per image, while scoring $w \in \mathbb{R}^P$7 candidate crops is $w \in \mathbb{R}^P$8 when efficient cell-average diagonal pooling is used.

3. Gradient-based IASM in aesthetic image captioning

Within ASE-MLLM, IASM is a lightweight, plug-in saliency extractor that explicitly extracts and encodes aesthetic saliency from an input image and routes it to IAS-ViT so that the downstream multimodal LLM can preferentially attend to regions with high aesthetic value during caption generation (Tao et al., 4 Sep 2025). The framework uses dynamic patch segmentation into $w \in \mathbb{R}^P$9 tiles, adjusts the number of tiles from $I$0 to $I$1 to support up to $I$2K resolution, and adds a thumbnail for global context. IASM outputs a scalar saliency map $I$3 at the target EAT layer’s spatial resolution and per-channel weighted activations that are later embedded as aesthetic saliency features.

The module combines an aesthetics-oriented backbone, EAT, with LayerCAM to compute aesthetic class activation maps. Given an image $I$4, the EAT model produces intermediate feature maps

$I$5

while a classifier head produces category scores

$I$6

with $I$7 denoting the top-scoring aesthetic category. LayerCAM defines

$I$8

and aggregates across channels as

$I$9

No ground-truth saliency annotations are required. The paper describes the resulting map as pseudo-saliency obtained by backpropagating gradients from the EAT classifier’s top-scoring aesthetic category prediction. Supervision comes solely from the downstream captioning objective.

IAS-ViT then fuses these saliency features with original image features through cross-attention. The encoder contains $H \times W$0 blocks, each using the standard “Norm → Self-Attention → Norm → MLP” structure with an added cross-attention block after self-attention. Let $H \times W$1 be the original image token sequence and $H \times W$2 be tokenized aesthetic saliency features. Cross-attention is defined by

$H \times W$3

$H \times W$4

The specialization is that queries come from aesthetic saliency features while keys and values come from original image features. The fused visual tokens are projected into the LLM’s text embedding space by an MLP projector composed of LayerNorm, Linear, and GELU layers, then concatenated with prompt tokens from InternLM2.Tokenizer. The fixed prompt is “Comment on this image from an aesthetic perspective.”

The training objective is standard next-token cross-entropy only:

$H \times W$5

No saliency alignment loss or auxiliary regularizer is introduced. The base model is InternVL2-8B with InternLM2-Chat, the optimizer is AdamW, the initial learning rate is $H \times W$6, weight decay is $H \times W$7, the warmup ratio is $H \times W$8, and the scheduler is cosine annealing. The paper does not specify whether EAT is frozen or updated during fine-tuning.

4. Saliency-augmented multi-pattern pooling as IASM for composition assessment

In SAMP-Net, the Saliency-Augmented Multi-pattern Pooling module can operate as an IASM for composition assessment by combining fixed composition-pattern masks, partition-wise visual pooling, and saliency augmentation (Zhang et al., 2021). The backbone is a ResNet18 pretrained on ImageNet. Input images are resized to $H \times W$9, producing a global feature map $F \in \mathbb{R}^{H \times W \times C}$0 of size $F \in \mathbb{R}^{H \times W \times C}$1 with $F \in \mathbb{R}^{H \times W \times C}$2 channels. An off-the-shelf unsupervised saliency detector, again the spectral residual method of Hou and Zhang (2007), produces a saliency map at $F \in \mathbb{R}^{H \times W \times C}$3; the paper notes that supervised saliency models were tried but were less effective.

The module analyzes composition through eight fixed patterns inspired by photographic composition: vertical symmetry, horizontal symmetry, diagonal band, anti-diagonal band, center composition, vertical symmetric tripartition, horizontal symmetric tripartition, and rule-of-thirds. For pattern $F \in \mathbb{R}^{H \times W \times C}$4, the spatial domain is split into non-overlapping partitions $F \in \mathbb{R}^{H \times W \times C}$5. With pixel-wise feature $F \in \mathbb{R}^{H \times W \times C}$6, average pooled partition features are

$F \in \mathbb{R}^{H \times W \times C}$7

Rather than saliency pooling, the corresponding saliency patch is reshaped into a vector

$F \in \mathbb{R}^{H \times W \times C}$8

so that spatial detail is preserved. The partition feature is

$F \in \mathbb{R}^{H \times W \times C}$9

Concatenating all partition features for pattern $1 \times 1$0 yields $1 \times 1$1, which is embedded by

$1 \times 1$2

with $1 \times 1$3 being ReLU.

Pattern weights are learned per image from the global feature map using global average pooling followed by a fully connected layer and softmax,

$1 \times 1$4

and aggregated as

$1 \times 1$5

This produces an interpretable pattern-weighted representation of composition. Two linear heads then split the representation into composition and attribute features,

$1 \times 1$6

and the Attentional Attribute Feature Fusion module learns

$1 \times 1$7

$1 \times 1$8

A distribution regression head then predicts a probability distribution over discrete composition scores $1 \times 1$9–$\mathbf{M} \in \mathbb{R}^{H \times W \times L},$0 by softmax, and the predicted mean score is the expectation under that distribution.

The paper additionally states that the same module can be expressed as a standalone IASM using saliency-weighted spatial integration within each pattern:

$\mathbf{M} \in \mathbb{R}^{H \times W \times L},$1

or, without normalization,

$\mathbf{M} \in \mathbb{R}^{H \times W \times L},$2

This variant is described as a straightforward way to deploy SAMP as a plug-in module in other aesthetic models.

5. Supervision, optimization, and task-specific outputs

The three IASM formulations differ markedly in supervision and objective design. In image cropping, supervision is provided by densely annotated crops on CPC and GAICD, with ranking pairs

$\mathbf{M} \in \mathbb{R}^{H \times W \times L},$3

using $\mathbf{M} \in \mathbb{R}^{H \times W \times L},$4 on CPC and $\mathbf{M} \in \mathbb{R}^{H \times W \times L},$5 on GAICD, and saliency maps computed by the spectral residual method for every training image (Tu et al., 2019). CPC contains $\mathbf{M} \in \mathbb{R}^{H \times W \times L},$6 images with $\mathbf{M} \in \mathbb{R}^{H \times W \times L},$7 crops per image; GAICD contains $\mathbf{M} \in \mathbb{R}^{H \times W \times L},$8 images with approximately $\mathbf{M} \in \mathbb{R}^{H \times W \times L},$9 crops per image. The default composition pattern is cross-rectangle with $L$0 partitions, and $L$1 is the default trade-off in the total loss.

In aesthetic captioning, supervision is purely textual. ASE-MLLM is trained and evaluated on DPC2022, PCCD, and RPCD, with splits following the original papers, and uses the standard next-token cross-entropy loss only (Tao et al., 4 Sep 2025). DPC2022 contains $L$2K images and more than $L$3M comments; PCCD contains $L$4 images and more than $L$5K critiques; RPCD contains $L$6K images and $L$7K comments. Evaluation uses BLEU-1..4, METEOR, ROUGE, CIDEr, SPICE, SPICE-L, Precision, and Recall. Saliency is therefore latent with respect to explicit annotation: it is generated by LayerCAM from the EAT classifier and optimized indirectly through captioning performance.

In composition assessment, supervision is distributional and multi-task. CADB contains $L$8 images after removing $L$9 biased images from $M_c \in \mathbb{R}^{h \times w}$00 real-world photos, and each image is rated by five professional fine-art raters on a $M_c \in \mathbb{R}^{h \times w}$01–$M_c \in \mathbb{R}^{h \times w}$02 scale (Zhang et al., 2021). The dataset is split into $M_c \in \mathbb{R}^{h \times w}$03 training images and $M_c \in \mathbb{R}^{h \times w}$04 test images. The composition distribution head is trained with squared EMD, and the paper extends this to weighted EMD to reduce content bias:

$M_c \in \mathbb{R}^{h \times w}$05

Attribute regression over rule of thirds, balancing elements, object emphasis, symmetry, and repetition uses mean squared error,

$M_c \in \mathbb{R}^{h \times w}$06

and the full objective is

$M_c \in \mathbb{R}^{h \times w}$07

A common misconception is that an IASM requires direct saliency supervision. The cited formulations do not support that generalization. The cropping model computes explicit saliency maps but trains the aesthetic score tensor through ranking plus saliency regularization; the captioning model requires no ground-truth saliency annotations at all; and the composition model uses an external saliency detector while optimizing composition distributions and attributes rather than saliency labels.

6. Empirical behavior, interpretability, and limitations

The cropping formulation reports strong results and extensive ablations (Tu et al., 2019). On CPC validation ranking accuracy, a $M_c \in \mathbb{R}^{h \times w}$08 grid achieves $M_c \in \mathbb{R}^{h \times w}$09, a $M_c \in \mathbb{R}^{h \times w}$10 grid $M_c \in \mathbb{R}^{h \times w}$11, a $M_c \in \mathbb{R}^{h \times w}$12 grid $M_c \in \mathbb{R}^{h \times w}$13, a $M_c \in \mathbb{R}^{h \times w}$14 grid $M_c \in \mathbb{R}^{h \times w}$15, multi-rectangle $M_c \in \mathbb{R}^{h \times w}$16, multi-direction $M_c \in \mathbb{R}^{h \times w}$17, and cross-rectangle with $M_c \in \mathbb{R}^{h \times w}$18 partitions $M_c \in \mathbb{R}^{h \times w}$19, which is the best and therefore the default. On the saliency-aware loss ablation, removing saliency-aware loss ($M_c \in \mathbb{R}^{h \times w}$20) leads to a $M_c \in \mathbb{R}^{h \times w}$21 drop relative to the best validation accuracy, and the best setting occurs at $M_c \in \mathbb{R}^{h \times w}$22. For best-crop prediction when trained on CPC, the model reaches IoU $M_c \in \mathbb{R}^{h \times w}$23 and Disp $M_c \in \mathbb{R}^{h \times w}$24 on FCDB, and IoU $M_c \in \mathbb{R}^{h \times w}$25 and Disp $M_c \in \mathbb{R}^{h \times w}$26 on FLMS. On multiple-crop ranking on GAICD with the $M_c \in \mathbb{R}^{h \times w}$27 split, it reports SRCC $M_c \in \mathbb{R}^{h \times w}$28, Acc_5 $M_c \in \mathbb{R}^{h \times w}$29, and Acc_10 $M_c \in \mathbb{R}^{h \times w}$30. Interpretability is provided through heatmaps of $M_c \in \mathbb{R}^{h \times w}$31 and $M_c \in \mathbb{R}^{h \times w}$32, which reveal content-important and composition-sensitive regions. Without $M_c \in \mathbb{R}^{h \times w}$33, the model learns a corner bias linked to dataset statistics; with $M_c \in \mathbb{R}^{h \times w}$34, $M_c \in \mathbb{R}^{h \times w}$35 concentrates more on visually salient regions.

The captioning formulation reports state-of-the-art performance across most metrics on mainstream AIC benchmarks and shows that IASC, defined as IASM plus IAS-ViT, consistently improves over fine-tuning alone (Tao et al., 4 Sep 2025). On DPC2022, the model reports BLEU-1 $M_c \in \mathbb{R}^{h \times w}$36, BLEU-2 $M_c \in \mathbb{R}^{h \times w}$37, BLEU-3 $M_c \in \mathbb{R}^{h \times w}$38, BLEU-4 $M_c \in \mathbb{R}^{h \times w}$39, METEOR $M_c \in \mathbb{R}^{h \times w}$40, ROUGE $M_c \in \mathbb{R}^{h \times w}$41, CIDEr $M_c \in \mathbb{R}^{h \times w}$42, SPICE $M_c \in \mathbb{R}^{h \times w}$43, SPICE-L $M_c \in \mathbb{R}^{h \times w}$44, Precision $M_c \in \mathbb{R}^{h \times w}$45, and Recall $M_c \in \mathbb{R}^{h \times w}$46. On PCCD, it reports BLEU-1 $M_c \in \mathbb{R}^{h \times w}$47, BLEU-2 $M_c \in \mathbb{R}^{h \times w}$48, BLEU-3 $M_c \in \mathbb{R}^{h \times w}$49, BLEU-4 $M_c \in \mathbb{R}^{h \times w}$50, METEOR $M_c \in \mathbb{R}^{h \times w}$51, ROUGE $M_c \in \mathbb{R}^{h \times w}$52, CIDEr $M_c \in \mathbb{R}^{h \times w}$53, SPICE $M_c \in \mathbb{R}^{h \times w}$54, SPICE-L $M_c \in \mathbb{R}^{h \times w}$55, Precision $M_c \in \mathbb{R}^{h \times w}$56, and Recall $M_c \in \mathbb{R}^{h \times w}$57. On RPCD, it reports BLEU-1 $M_c \in \mathbb{R}^{h \times w}$58, BLEU-2 $M_c \in \mathbb{R}^{h \times w}$59, BLEU-3 $M_c \in \mathbb{R}^{h \times w}$60, BLEU-4 $M_c \in \mathbb{R}^{h \times w}$61, METEOR $M_c \in \mathbb{R}^{h \times w}$62, ROUGE $M_c \in \mathbb{R}^{h \times w}$63, CIDEr $M_c \in \mathbb{R}^{h \times w}$64, SPICE $M_c \in \mathbb{R}^{h \times w}$65, SPICE-L $M_c \in \mathbb{R}^{h \times w}$66, Precision $M_c \in \mathbb{R}^{h \times w}$67, and Recall $M_c \in \mathbb{R}^{h \times w}$68. The paper notes that CIDEr and SPICE on RPCD lag the best traditional methods and attributes this to a distribution mismatch between RPCD language style and the pretrained LLM. Qualitative analyses show that aesthetic saliency maps include background elements such as reflections, leaves, sky, and clouds, and that captions shift toward composition, warm colors, reflections, and placement rather than simply naming objects.

The composition-assessment formulation reports that SAMP-Net performs more favorably than prior aesthetic assessment approaches on CADB (Zhang et al., 2021). With a ResNet18 backbone for fairness, SAMP-Net achieves MSE $M_c \in \mathbb{R}^{h \times w}$69, EMD $M_c \in \mathbb{R}^{h \times w}$70, SRCC $M_c \in \mathbb{R}^{h \times w}$71, and LCC $M_c \in \mathbb{R}^{h \times w}$72, compared with A-Lamp at MSE $M_c \in \mathbb{R}^{h \times w}$73, EMD $M_c \in \mathbb{R}^{h \times w}$74, SRCC $M_c \in \mathbb{R}^{h \times w}$75, and LCC $M_c \in \mathbb{R}^{h \times w}$76. Ablations show that weighted EMD improves EMD from $M_c \in \mathbb{R}^{h \times w}$77 to $M_c \in \mathbb{R}^{h \times w}$78 and SRCC from $M_c \in \mathbb{R}^{h \times w}$79 to $M_c \in \mathbb{R}^{h \times w}$80; adding multi-pattern pooling improves SRCC from $M_c \in \mathbb{R}^{h \times w}$81 to $M_c \in \mathbb{R}^{h \times w}$82; learning pattern weights improves SRCC from $M_c \in \mathbb{R}^{h \times w}$83 to $M_c \in \mathbb{R}^{h \times w}$84; adding saliency improves SRCC from $M_c \in \mathbb{R}^{h \times w}$85 to $M_c \in \mathbb{R}^{h \times w}$86 and LCC from $M_c \in \mathbb{R}^{h \times w}$87 to $M_c \in \mathbb{R}^{h \times w}$88; and the full model with saliency, attributes, and attentional fusion achieves the best overall scores. The learned pattern weights reveal dominant compositional patterns per image, including rule-of-thirds, symmetry, and center composition.

The limitations are similarly task-specific. In cropping, residual spatial bias may persist, excessively large $M_c \in \mathbb{R}^{h \times w}$89 reduces model capacity by suppressing $M_c \in \mathbb{R}^{h \times w}$90 broadly, and fine-grid approximation of non-grid patterns may introduce boundary artifacts (Tu et al., 2019). In captioning, challenges arise from semantic diversity, colloquial versus formal language styles in critiques, incomplete coverage of aesthetic aspects in annotations, and possible failures on images whose beauty depends on subtle artistic intent not captured by EAT’s categories or LayerCAM gradients (Tao et al., 4 Sep 2025). In composition assessment, the model can underestimate some high-scoring images, often due to distracting backgrounds or complex patterns, and the paper identifies content bias as a core dataset issue that motivates weighted EMD (Zhang et al., 2021). Taken together, these results indicate that IASM-style modules are effective when saliency is aligned with aesthetic structure, but that they remain sensitive to dataset bias, annotation style, and the granularity of the aesthetic prior they encode.