Visual Interestingness in Image Analysis

• Visual interestingness is a concept defining why an image attracts attention by combining unexpected elements with contextual relevance.
• Research employs pairwise comparisons, classification metrics, and novelty detection to gauge interestingness, revealing nuances in human and model judgments.
• Computational paradigms, from hand-crafted feature fusion to deep feature rarity measures, provide actionable insights for image selection and robotic applications.

Searching arXiv for recent and foundational papers on visual interestingness, memorability, saliency, and related formulations. Visual interestingness is a family of concepts used to describe why an image, region, scene, label, or visual episode is worth attending to, selecting, communicating, or revisiting. In current research, it is not a single scalar property with a universally accepted definition. It has been defined as the ability of an image to attract and hold attention, as novelty or unusualness relative to previously seen data, as a binary interesting-versus-uninteresting judgment, as the informativeness of an annotation for a listener with prior knowledge, and, more abstractly, as a composition of unexpectedness and relevance (Abdullahu et al., 15 Oct 2025, Wagstaff et al., 2018, Bracha et al., 2018, Exman, 2014). This suggests that visual interestingness is best understood as a task-dependent construct whose operational meaning changes with the unit of analysis and the intended downstream use.

1. Conceptual scope

A prominent contemporary definition describes visual interestingness as what attracts and holds attention. In that formulation, the practical question is whether a model can reproduce human judgments about which of two images is more interesting, rather than whether it can assign a stable absolute score to a single image (Abdullahu et al., 15 Oct 2025). A different but compatible tradition treats interestingness as novelty, unusualness, or anomaly in an image collection, with the additional requirement that the reason for selection be interpretable in human-understandable visual terms (Wagstaff et al., 2018).

A more general theoretical account states that interestingness is bipolar and arises from exactly two functions: unexpectedness and relevance. In that paradigm, the formal statement is

$\text{Interestingness} = R \circ U,$

where $U$ denotes unexpectedness and $R$ denotes relevance (Exman, 2014). In visual settings, this maps naturally onto scenes or regions that both deviate from what is expected and remain pertinent to the current semantic or task context. This suggests that purely salient but irrelevant content, and purely relevant but unsurprising content, are both incomplete cases.

Research on annotation selection extends the concept further by relocating interestingness from the image alone to the interaction between image content and a listener model. There, a label is interesting if it reduces uncertainty over the space of possible labels for a prospective listener, so interestingness becomes semantic and communicative rather than purely perceptual (Bracha et al., 2018).

2. Operationalizations and measurement

The empirical study of visual interestingness uses several incompatible but technically precise measurement regimes. A useful distinction is between image-level preference, image-level classification, label-level informativeness, and stream-level online selection.

Unit of analysis	Annotation criterion	Representative formalization
Image pairs	“Which image is more interesting to you?”	$A^{(M,N)}(\mathcal{S})$ (Abdullahu et al., 15 Oct 2025)
Single images	Interesting vs. uninteresting	ACC, ROC, AUC (Sun et al., 2019)
Candidate labels	Uncertainty reduction for a listener	$cw(i)$ (Bracha et al., 2018)
Robotic streams	Online precision of interesting frames	AUC-OP (Wang et al., 2021)

A central methodological result is that single-image interestingness is often too ambiguous to be discriminative. In a pairwise study on 1,000 everyday Flickr images arranged into 2,500 image pairs, human consensus held for only 56.3% of pairs, while GPT-4o was 95.5% self-consistent. Human–GPT-4o agreement was 66.2% overall, rising to 73.8% on the human consensus subset $\mathcal{C}_H$ and falling to 56.5% on the dissent subset $\mathcal{D}_H$. The agreement metric was defined as

$$A^{(M,N)}(\mathcal{S}) := \frac{1}{|\mathcal{S}|}\sum_{i=1}^{|\mathcal{S}|}\mathbb{I}(y_M^{(i)} = y_N^{(i)}),$$

and these results were taken as evidence of partial alignment rather than identity between human and model judgments (Abdullahu et al., 15 Oct 2025).

Binary image interestingness has also been treated as a conventional supervised classification problem. On the 2016 Predicting Multimedia Interestingness Task dataset, comprising 7,396 middle frames from shots in 78 movie trailers, performance was reported using accuracy and AUC. The strongest configuration combined cue-specific feature fusion with multiple kernel learning and achieved ACC $=0.944$ and AUC $=0.999$ (Sun et al., 2019).

At the semantic-description level, informativeness is measured by entropy reduction. For a candidate label $U$0, the confidence-weighted score is

$U$1

where the listener’s prior over the joint label space is approximated by a Chow-Liu tree or a mixture of such trees (Bracha et al., 2018). Here, interestingness is not about whether the label is visually obvious, but whether communicating it changes what a listener can infer.

In robotic streams, ordinary static precision or ROC analysis is insufficient because future frames must not leak into the decision rule. For that setting, AUC-OP was introduced to evaluate online precision under a top-$U$2-style constraint over prefixes of the observed sequence (Wang et al., 2021).

3. Computational paradigms

One major line of work models interestingness through hand-crafted or fused cues. A unified binary-interestingness framework organized cues into unusualness, aesthetics, and general preferences. Within-cue redundancy was reduced by discriminant correlation analysis or multi-set discriminant correlation analysis, and cue-level heterogeneity was handled by SimpleMKL. In the final kernel combination,

$U$3

general preferences contributed the largest weight. Unusualness alone was weak, with ACC $U$4 and AUC $U$5, whereas aesthetics reached ACC $U$6, AUC $U$7, and general preferences reached ACC $U$8, AUC $U$9 (Sun et al., 2019). This indicates that, in that benchmark, interestingness was driven more by broad preference and aesthetic structure than by outlierness alone.

A second paradigm treats interestingness as rarity in deep feature space. DeepRare2019 uses convolutional feature maps from VGG16, computes histogram-based rarity by

$R$0

backprojects rarity to each map, fuses within and across layer groups, and adds a face-related activation from feature map #105 in layer 15. The method requires no training, runs in less than a second per image on CPU only, and is reported to be always in the top-3 models on all tested datasets and metrics, with no other model exhibiting the same regularity and genericity (Mancas et al., 2020). In this formulation, interestingness is close to bottom-up surprise, albeit supplemented by one explicit top-down cue.

A third paradigm treats interestingness as novelty under an incrementally updated model of what is already known. DEMUD selects the image with maximal reconstruction error,

$R$1

using CNN feature vectors rather than raw pixels. The residual

$R$2

functions as the explanation of why the image was selected, and an up-convolutional network visualizes both expected and novel content (Wagstaff et al., 2018). This approach ties interestingness directly to discovery, non-redundancy, and interpretability.

Large multimodal models introduce a fourth paradigm: pairwise preference imitation plus distillation. GPT-4o-generated pair labels were used to train a Siamese CLIP-based ranker with pairwise score

$R$3

After training for 25 epochs and repeating 50 times on half-splits of the dataset, the learned ranker achieved test accuracy around 84.8% when trained on GPT-4o labels and around 77.5% when trained on human labels, with global rank correlation about 0.59 in both cases (Abdullahu et al., 15 Oct 2025). The result is not that model judgments equal human judgments, but that model-produced pairwise supervision is sufficiently structured to distill into a compact ranking function.

4. Relation to aesthetics, memorability, appeal, complexity, and popularity

Visual interestingness is frequently conflated with aesthetics, but the two are not interchangeable. Image Content Appeal Assessment (ICAA) explicitly separates the appeal of depicted content from Image-Aesthetics Assessment. On food and room datasets, the reported PLCC and SRCC correlations between appeal labels and three aesthetics baselines were small or near zero, and a user study found more than 76% preference for appeal-enhanced images, specifically 76.53% for food and 82.74% for rooms (Chen et al., 2024). In that formulation, visual interestingness is closer to “positive interest” in the subject matter than to photographic craft.

Interestingness is also distinct from memorability. In the pairwise everyday-image study, memorability was the weakest correlate among the compared predictors, below aesthetic and common-interest models (Abdullahu et al., 15 Oct 2025). In scientific visualization, interestingness was measured directly by the question “Is the visualization interesting?” on a 7-point Likert-scale, yet its Spearman correlation with memorability was $R$4, and the paper stated that happiness and interestingness are not statistically correlated with the memorability of scientific visualizations (Li et al., 2018). By contrast, memorability has been characterized as “an image-computable measure of information utility” (Bylinskii et al., 2021). A plausible implication is that memorable content and interesting content overlap only partially because one concerns later recognition and the other concerns immediate engagement or selection.

The relation to beauty and statistical structure is also nontrivial. In synthetic abstract images, preference peaked at intermediate entropic complexity, and coarse-grained algorithmic complexity $R$5, termed structural complexity, was proposed as a better predictor than raw entropic measures. The paper states that “Structural complexity, in a sense, measures noiseless entropic complexity or interestingness,” and reports a reversed-U preference curve rather than monotonic preference for either order or noise (Lakhal et al., 2019). This suggests that some forms of visual interestingness depend on organized complexity rather than maximal disorder.

Popularity is a further confounder. Flickr beauty judgments correlated with popularity at about $R$6, but popularity and intrinsic visual quality were explicitly treated as distinct. A supervised aesthetics model trained on Flickr crowd judgments retrieved low-popularity photos whose median perceived beauty equaled that of the most popular photos, with average beauty lower by only 1.5% (Schifanella et al., 2015). Social attention therefore cannot be taken as a direct measure of visual interestingness, even in image-sharing platforms.

5. Domain-specific forms and applications

In multimedia retrieval, interestingness prediction has been framed as late fusion over heterogeneous predictors. On the ImageCLEFfusion 2022 task, 29 provided inducer scores were combined by

$R$7

with weights optimized under an MSE objective. Equal-weight fusion reached MAP@10 $R$8, while PSO-weighted fusion and TNA-weighted fusion both reached 0.109 (Shoukat et al., 2022). The absolute values remained low, which the paper interpreted as evidence that media interestingness prediction is intrinsically difficult.

Scientific visualization shows a domain where interestingness must be separated from memorability and from generic design intuitions. A curated corpus of 1,142 SciVis images, with memorability scores computed for 228 images, was evaluated using objective measures, subjective ratings, and sentiment ratings. Clutter and number of distinct colors were more strongly associated with memorability than interestingness was, and the negative correlation between clutter and memorability was $R$9 (Li et al., 2018). In this domain, being “interesting” does not imply being memorable.

Robotics introduces a strongly temporal and adaptive notion. Unsupervised online learning for robotic interestingness defines a scene as interesting when it is novel, worth attention, or useful for exploration, but allows that judgment to decay as similar scenes recur. The system uses a three-stage architecture for long-term, short-term, and online learning, together with a translation-invariant 4-D visual memory. The method reports about 20% higher accuracy than state-of-the-art unsupervised methods in subterranean tunnel environments and online runtime of about 14.58 ms/frame in the journal version (Wang et al., 2021). In a related formulation, visual memorability was used as a proxy: interestingness was treated as negatively correlated with memory reading confidence (Wang et al., 2020).

Human-interactive robotics adds a second layer in which novelty-based scene selection is refined by mission-specific object relevance. AirInteraction first uses online unsupervised interestingness recognition onboard the robot, transmits only potential interesting scenes to a base station, and then incorporates human-drawn bounding boxes into a few-shot object detector based on TFA. The reported communication reduction ranged from 80% to 91%, and weighted novel-sample reuse with $A^{(M,N)}(\mathcal{S})$0 or $A^{(M,N)}(\mathcal{S})$1 outperformed TFA on most datasets (Kim et al., 2022). Here, visual interestingness is both what the robot has not yet assimilated and what a human operator declares mission-relevant.

Explainable reinforcement learning uses yet another domain-specific decomposition. Interestingness elements are mined from an agent’s history along four dimensions: frequency, execution certainty, transition-value, and sequence. These are turned into short visual summaries of behavior. The user study reported that the diversity of aspects captured by these elements was crucial for helping humans understand an agent’s capabilities and limitations (Sequeira et al., 2019). In this context, visual interestingness is less a property of the rendered frame than a criterion for selecting explanatory episodes.

6. Ambiguities, biases, and unresolved questions

A recurring issue is that interestingness is subjective but structured. The pairwise Flickr study found that single-image yes/no judgments saturate because most everyday images are deemed interesting, whereas pairwise ranking exposes preference structure and disagreement (Abdullahu et al., 15 Oct 2025). This implies that annotation format is not a neutral design choice; it changes what aspect of interestingness becomes measurable.

Model-based prediction introduces additional biases. GPT-4o exhibited a systematic order bias in pairwise inputs, with only 64% of annotations unchanged after swapping image order, and later experiments filtered to the subset without this bias. Qualitative analysis also found recurring explanation themes such as cute/emotional, uniqueness, vibrant colors, and action/dynamics, some shared with humans and some more model-specific (Abdullahu et al., 15 Oct 2025). Thus, partial alignment is not merely reduced accuracy; it includes systematic differences in what cues are treated as interesting.

Population-level listener models are another source of abstraction. The entropy-reduction framework for informative annotations relies on a learned prior over label co-occurrences, approximated by a Chow-Liu tree or mixture of trees, and assumes classifier confidences are usable as weights (Bracha et al., 2018). This makes the method tractable and effective, but it also means that interestingness is computed relative to a corpus-derived listener rather than an individualized expert or user.

Robotic definitions expose a different limitation: general interestingness in unknown, unstructured environments is not identical to task-specific interestingness in domains such as household robotics. The unsupervised online framework explicitly notes that its emphasis on general interestingness may not directly fit settings where “interesting” is user-defined or mission-specific (Wang et al., 2021). Human-in-the-loop extensions address this only partially by adding few-shot object supervision after novelty filtering (Kim et al., 2022).

Finally, explanation itself has a bandwidth problem. In explainable reinforcement learning, a summary containing one highlight from every interestingness element did not yield the best understanding and often lowered user confidence, indicating that too much heterogeneous evidence can impede coherent interpretation (Sequeira et al., 2019). This suggests that visual interestingness is not only a property to be predicted; it is also a selection principle that must be balanced against cognitive load.

Taken together, these results support a restrained conclusion. Visual interestingness is not reducible to saliency, aesthetics, memorability, popularity, or anomaly detection, although each can supply one of its operational faces. What persists across formulations is a common emphasis on selective value: content becomes interesting when it changes attention, preference, knowledge, or action in a way that is not captured by raw relevance alone.