Context Sensitivity Improves Human-Machine Visual Alignment

Abstract: Modern machine learning models typically represent inputs as fixed points in a high-dimensional embedding space. While this approach has been proven powerful for a wide range of downstream tasks, it fundamentally differs from the way humans process information. Because humans are constantly adapting to their environment, they represent objects and their relationships in a highly context-sensitive manner. To address this gap, we propose a method for context-sensitive similarity computation from neural network embeddings, applied to modeling a triplet odd-one-out task with an anchor image serving as simultaneous context. Modeling context enables us to achieve up to a 15% improvement in odd-one-out accuracy over a context-insensitive model. We find that this improvement is consistent across both original and "human-aligned" vision foundation models.

• The paper introduces a context-sensitive similarity computation method that adapts visual embeddings based on contextual cues to enhance odd-one-out decision accuracy.
• It employs an affine transformation and a neural network module to modulate pairwise image similarity, achieving up to a 15% improvement over context-insensitive baselines.
• The approach dynamically reshapes embedding spaces to mirror human perceptual judgments, offering practical gains and theoretical insights into visual cognition.

Context-Sensitive Modeling for Enhanced Human-Machine Visual Alignment

Motivation and Background

Traditional vision foundation models (FMs) such as DINOv2, SigLIP, and ViT represent images as fixed points in high-dimensional embedding spaces, optimizing for tasks like object classification under the assumption of context-invariant similarity. This paradigm diverges fundamentally from human perceptual and cognitive processes, which are highly context-sensitive and dynamically adapt object representations based on environmental or task-specific cues. Contextual adaptation is crucial for modeling behavior and similarity judgments as evidenced by asymmetries and context dependencies in human similarity assessments [tversky1977features]. Addressing the gap between static FM embeddings and dynamic human representations, this paper introduces a method for context-sensitive similarity computation that conditions visual similarity on an anchor image serving as context.

Model Architecture

The presented approach transforms triplet odd-one-out tasks by incorporating context into similarity computation. Each triplet consists of three images $\{p, q, r\}$ and a context image $c$. Images are encoded using FMs, followed by an affine transformation and normalization to promote human alignment. The context-sensitive module applies a neural network that maps the context embedding to a low-rank matrix, which is used as a kernel for modulating pairwise image similarity. Formally, for embeddings $\tilde{x}_i$ (image $i$) and $A_c$ (context-dependent matrix), the context-sensitive similarity is $s_{i,j|c} = \tilde{x}_i^\top A_c \tilde{x}_j$. Odd-one-out decisions assign the most similar pair, conditioning on the context, as the in-group; the oddball is selected accordingly.

Figure 1: Qualitative comparison showing context-sensitive and context-insensitive model operations; context-sensitive models predict human odd-one-out choices by modulating similarity based on context, reflected in representational similarity matrices and embedding space projections.

Quantitative Evaluation

Experiments utilize the ImageNet-HSJ dataset restructured for triplet-with-context evaluation, enabling direct comparison with human judgments. The main quantitative result is that context-sensitive models consistently outperform both FM baselines and context-insensitive transformations across all tested architectures and their human-aligned variants. For instance, context-sensitive variants achieve up to a 15% improvement in odd-one-out accuracy relative to context-insensitive counterparts. Paired bootstrap analyses reveal that these improvements are statistically significant with all confidence intervals excluding zero.

Figure 2: Odd-one-out accuracy for context-sensitive models versus baselines; context-sensitive modeling delivers consistent accuracy improvements across supervised, contrastive, and self-supervised FM architectures.

Analysis of full results demonstrates that, for each FM, both context-insensitive and context-sensitive transforms increase alignment with human judgments, with the latter providing the largest effect size. Gains are robust for both original and human-aligned embeddings, indicating that context-sensitive similarity computation can be layered atop existing human alignment techniques to further improve visual judgment prediction.

Qualitative Analysis of Representation Spaces

Qualitative assessments employ representational similarity matrices (RSMs) and principal component analysis (PCA) to visualize embedding space modulation. In triplet tasks, context-sensitive models dynamically re-weight features, pushing ambiguous objects (e.g., horse-cart) toward appropriate clusters (animal or vehicle) depending on the context image, paralleling human reclassification. This modulation results in embedding space restructurings that increase similarity among contextually relevant items and reduce similarity among irrelevant items.

Figure 3: Context-induced reshaping of embedding space visualized via RSMs and PCAs; context-sensitive models cause dynamic cluster shifts based on contextual cues, matching human judgments.

Implications and Future Directions

The work provides strong empirical evidence that context-sensitive embedding modulation enhances model alignment with human visual similarity judgments, both in terms of accuracy and semantic structure. The methodology is psychologically plausible, inspired by foundational cognitive theories such as Tversky’s feature-based similarity, and offers an explicit path to integrating contextual information at the representational level in FMs.

Practical implications include improved deployment of FMs for tasks requiring nuanced similarity judgments (e.g., autorating, human-in-the-loop annotation, visual reasoning in complex environments). Theoretical implications extend to expanding representational alignment frameworks beyond static transformation, allowing dynamic adaptation and attention-shifting in embedding spaces.

Future research can generalize context-sensitive mechanisms to sequential dependencies, intrinsic goals, and vision-LLMs for multi-image reasoning, as well as asymmetric similarity tasks where reference-dependency is essential. Additionally, further analysis of embedding space dynamics in the presence of diverse contextual and motivational states may yield more robust models of human cognition.

Conclusion

Context-sensitive similarity computation enables foundation models to more closely approximate human visual alignment, significantly improving odd-one-out prediction accuracy and producing semantically coherent representational modulations. Layered with human-aligned transformations, context-sensitive models generalize across multiple FM architectures and yield representation spaces that dynamically reflect context-specific relevance. This paradigm will likely influence future approaches to visual cognition modeling, context-aware vision systems, and enhanced interpretability of similarity-driven downstream tasks in artificial intelligence.