Visual Semantic Salience Modeling

• Visual Semantic Salience (VSS) is a framework that quantifies the importance of semantic elements in visual scenes by fusing semantic content with contrast cues.
• It employs dual-pathway architectures like SCA/SCAFI that combine CNN-derived semantic maps with contrast-based self-information for enhanced fixation prediction.
• The approach further uses Attention Graph encoding to aggregate object-level scanpaths, improving scanpath agreement and cognitive assessments in tasks like age classification and ASD screening.

Visual Semantic Salience (VSS) denotes the quantification and modeling of the importance of semantic entities within visual scenes, as inferred from human fixations and gaze behavior. Unlike purely pixel-based or low-level saliency, VSS identifies which objects or regions are deemed attention-worthy due to both their semantic content (e.g., faces, text, meaningful objects) and their context-dependent contrast. Computationally, VSS is realized through models that fuse semantic understanding and contrast detection or, more recently, via data structures—such as the Attention Graph—that embed both spatial saliency and temporal gaze dynamics at the level of annotated objects or semantic attributes. This dual approach bridges traditional saliency maps and high-level behavioral analysis, supporting both robust fixation prediction and cognitive-state assessment.

1. Mathematical Formulations of Visual Semantic Salience

VSS is formalized in two major computational paradigms: (a) pixel-space fusion of semantic- and contrast-aware cues and (b) scene-graph encoding of object-level attention.

1.1 Pixel-Space: Dual-Pathway Saliency (SCA/SCAFI Models)

The integrated VSS model consists of two additive components: semantic-aware saliency ($S_{\mathrm{sem}}$) and contrast-aware saliency ($S_{\mathrm{cont}}$), with final prediction

$$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$

where $N_{\max}$ denotes Maxima Normalization.

• Semantic-Aware Saliency ($S_{\mathrm{sem}}$): Two methods are proposed:
  • End-to-End CNN (SCA): $$S_{\mathrm{sem}}(I) = f_{\mathrm{sem}}(I;\theta_{\mathrm{sem}})$$, with $f_{\mathrm{sem}}$ a lightweight VGG-16-based network trained to minimize MSE between predicted map and Gaussian-blurred fixation density.
  • Feature-Integration SAS (SCAFI): Extract multiscale VGG-16 features, compute per-feature saliency $P_{l,k}(x,y)$ via softmax over maximal activations, aggregate by learned/fixed weights across layers:

  $$S_{\mathrm{sem}}(x,y) = \sum_{l=1}^{5} w_l S_l(x,y),$$

  with $S_l(x,y)$ the layerwise sum and $S_{\mathrm{cont}}$0 the pooling weights.

• Contrast-Aware Saliency ($S_{\mathrm{cont}}$1): At multiple scales $S_{\mathrm{cont}}$2, extract patch matrix $S_{\mathrm{cont}}$3 and learn ICA dictionary $S_{\mathrm{cont}}$4. The self-information per patch is given by

$S_{\mathrm{cont}}$5

where $S_{\mathrm{cont}}$6, and $S_{\mathrm{cont}}$7 is the empirical marginal distribution.

• Normalization and Fusion: Each map is Maxima Normalized prior to summation:

$S_{\mathrm{cont}}$8

where $S_{\mathrm{cont}}$9 and $$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$0 denote sum and count of significant local maxima, respectively (Sun, 2018).

1.2 Object-Level: Attention Graph Encoding

An image with $$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$1 annotated semantic objects $$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$2 yields an Attention Graph $$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$3, where:

• Nodes $$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$4 represent objects.

• Directed, weighted edges $$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$5 represent transition probabilities between objects, defined as:

$$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$6

for $$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$7 observers with semantic scanpaths $$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$8 (Yang et al., 11 Mar 2025).

Each node also carries a normalized fixation density: $$S_{\mathrm{final}}(x,y) = N_{\max}(S_{\mathrm{sem}})(x,y) + N_{\max}(S_{\mathrm{cont}})(x,y),$$9 possibly smoothed via a Gaussian.

2. Pathway Architectures and Implementation

2.1 SCA (End-to-End Network)

• Layers: VGG-16 conv1–conv3 (fine-tuned), pooling, 6 conv layers, upsampling via deconvolution to $N_{\max}$0.

• Training: SGD, batch size 2, 24K iterations, with L2-weight decay; learning rate halved every 100 iterations; MSE loss against fixation densities.

2.2 SCAFI (Dynamic Feature Integration)

• SAS: VGG forward pass, no further learning, linear pooling of 5 scales.

• CAS: Offline ICA dictionary learning; online filter response and histogram-based self-information.

• No additional supervised learning past VGG and ICA unsupervised stages (Sun, 2018).

2.3 Attention Graph Construction

• Semantic scanpath encoding collapses consecutive fixations on the same object to remove spatial jitter.

• Supports object-level (SemScan(obj)) and attribute-level (SemScan(att)) aggregations, enabling hierarchical VSS analysis (Yang et al., 11 Mar 2025).

3. Evaluation Metrics and Benchmarking

• Shuffled AUC (sAUC): Primary for pixel-space models, evaluating true positive against shuffled negatives to reduce central bias (Sun, 2018).

• Attention Graph Metrics:

  • Transition-Consistency Score:

  $N_{\max}$1

  where $N_{\max}$2 is a semantic scanpath (Yang et al., 11 Mar 2025). - Saliency-Weighted Score:

  $N_{\max}$3

• Datasets: Bruce, Cerf, ImgSal, Judd, PASCAL-S (for SCA/SCAFI), with SALICON for SCA training; SALIENCY4ASD and developmental datasets for Attention Graph experiments.

Metric	SCAFI	DPN	Deep Models	Saliency4ASD (Graph)
sAUC (avg.)	0.7238	0.689	0.69–0.74	–
ASD Screening Accuracy (%)	–	–	–	up to 93
Age Classif. Accuracy (%)	–	–	–	80

4. Empirical Findings and Functional Properties

• Performance: SCAFI achieves average sAUC ≈ 0.7238, outperforming contemporary deep models (DPN, DeepGazeII, SALICON) and strong classical methods (AWS, SGP).

• Speed: SCAFI runs ≈ 70× faster than SALICON, with no further supervised training beyond initial pretraining.

• Ablation: Fusion of semantic and contrast pathways yields significant gains (CAS alone ≈ 0.6902 sAUC; SAS alone ≈ 0.7077; fusion ≈ 0.7238).

• Plausibility: SCAFI accurately detects pop-out and high-contrast elements frequently missed by end-to-end CNNs given semantic pretraining alone (Sun, 2018).

• Attention Graph:

  • Reduces inter-observer jitter by collapsing fixations into semantic clusters.
  • Within- and between-observer scanpath agreement rises to ≈ 0.67 (object) and ≈ 0.79 (attribute) using $N_{\max}$4 metric, compared to ≈ 0.39–0.43 with classic sequence metrics.
  • Demonstrates robust discrimination in age and ASD cognitive-state tasks, rivaling or surpassing CNN-based alternatives despite no pixel-level learning (Yang et al., 11 Mar 2025).

5. Semantic Hierarchies and Cognitive-State Applications

The hierarchical structure of the Attention Graph enables VSS analysis at multiple semantic resolutions:

• Object-level: Direct mapping from fixations to objects, minimizing within-object spatial noise.
• Attribute-level: Objects grouped by shared high-level attributes (e.g., "watchability"), facilitating higher-order attention modeling.

Applications extend to:

• Cognitive-State Assessment: Age classification (18 vs 30 months) achieves 80% accuracy; ASD screening up to 93% accuracy via attention pattern analysis, all without visually learned features—performance reached solely by VSS-driven scanpath analysis.
• Transparent Evaluation: Permits object- and attribute- grounded benchmarking for both human and model-generated scanpaths (Yang et al., 11 Mar 2025).

6. Comparative Analysis and Model Properties

VSS models, whether constructed as fusion saliency maps or object-centric graphs, provide several advantages over classic fixation prediction:

• Explicit modeling of long-term (semantic) and short-term (contrast) biases in attention allocation.
• Transparent, interpretable frameworks for visual cognition, supporting both psychophysical plausibility tests and practical clinical scenarios.
• The Attention Graph formalism directly represents both spatial salience and scanpath statistics in a unified mathematical object, making it uniquely suited for empirical and diagnostic tasks.

A plausible implication is that future progress in VSS will continue to leverage both high-capacity deep visual representations and explicit semantic/object-level abstractions, enabling deeper insight into both collective and individual viewing strategies.