Few-shot Personalized Saliency Prediction Based on Inter-personnel Gaze Patterns (2307.02799v3)
Abstract: This paper presents few-shot personalized saliency prediction based on inter-personnel gaze patterns. In contrast to general saliency maps, personalized saliecny maps (PSMs) have been great potential since PSMs indicate the person-specific visual attention useful for obtaining individual visual preferences. The PSM prediction is needed for acquiring the PSMs for unseen images, but its prediction is still a challenging task due to the complexity of individual gaze patterns. Moreover, the eye-tracking data obtained from each person is necessary to construct and predict PSMs, but it is difficult to acquire the massive amounts of such data. One solution for realizing PSM prediction from the limited amount of data is the effective use of eye-tracking data obtained from other persons. To efficiently treat the PSMs of other persons, this paper focuses on the selection of images to acquire eye-tracking data and the preservation of structural information of PSMs of other persons. In the proposed method, such images are selected such that they bring more diverse gaze patterns to persons, and the structural information is preserved by adopting the tensor-based regression method. Experimental results demonstrate that the above two points are beneficial for the few-shot PSM prediction.
- “A model of saliency-based visual attention for rapid scene analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1254–1259, 1998.
- “A brief survey of visual saliency detection,” Multimedia Tools and Applications, vol. 79, pp. 34605–34645, 2020.
- “Salient object detection: A survey,” Computational Visual Media, vol. 5, pp. 117–150, 2019.
- “Graph-based visual saliency,” in Proc. Advances in Neural Information Processing Systems, 2007, pp. 545–552.
- “Image signature: Highlighting sparse salient regions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 34, no. 1, pp. 194–201, 2012.
- “Salgan: Visual saliency prediction with generative adversarial networks,” arXiv preprint arXiv:1701.01081, 2017.
- “Contextual encoder–decoder network for visual saliency prediction,” Neural Networks, vol. 129, pp. 261–270, 2020.
- “Personalized visual saliency: Individuality affects image perception,” IEEE Access, vol. 6, pp. 16099–16109, 2018.
- “Curious eyes: Individual differences in personality predict eye movement behavior in scene-viewing,” Cognition, vol. 122, no. 1, pp. 86–90, 2012.
- “Visual attention saccadic models learn to emulate gaze patterns from childhood to adulthood,” IEEE Trans. Image Processing, vol. 26, no. 10, pp. 4777–4789, 2017.
- “Beyond universal saliency: Personalized saliency prediction with multi-task cnn,” in Proc. Int’l Joint Conf. Artificial Intelligence, 2017, pp. 3887–3893.
- “Where’s your focus: Personalized attention,” arXiv preprint arXiv:1802.07931, 2018.
- “Few-shot personalized saliency prediction using person similarity based on collaborative multi-output gaussian process regression,” in Proc. IEEE Int’l Conf. Image Processing, 2021, pp. 1469–1473.
- “Learning from unique perspectives: User-aware saliency modeling,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, 2023, pp. 2701–2710.
- “Predicting personalized saliency map for people with autism spectrum disorder,” in Proc. International Conf. Content-based Multimedia Indexing, 2023, pp. 34–40.
- “Personalized saliency and its prediction,” IEEE Trans. Pattern Analysis and Machine Intelligence, pp. 2975–2989, 2018.
- “The interestingness of images,” in Proc. IEEE Int’l Conf. Computer Vision, 2013, pp. 1633–1640.
- “Towards measuring and inferring user interest from gaze,” in Proc. Int’l Conf. World Wide Web Companion, 2017, pp. 525–533.
- “Personalized video summarization based exclusively on user preferences,” in Proc. European Conf. Information Retrieval, 2020, pp. 305–311.
- “Multiple pairwise ranking networks for personalized video summarization,” in Proc. IEEE/CVF International Conf. Computer Vision, 2021, pp. 1718–1727.
- Xi Yin and Xiaoming Liu, “Multi-task convolutional neural network for pose-invariant face recognition,” IEEE Trans. Image Processing, vol. 27, no. 2, pp. 964–975, 2017.
- “Few-shot personalized saliency prediction based on adaptive image selection considering object and visual attention,” Sensors, vol. 20, no. 8, pp. 2170, 2020.
- Eric F Lock, “Tensor-on-tensor regression,” Journal of Computational and Graphical Statistics, vol. 27, no. 3, pp. 638–647, 2018.
- “Collaborative multi-output gaussian processes.,” in Proc. Association for Uncertainty in Artificial Intelligence, 2014, pp. 643–652.
- “Yolox: Exceeding yolo series in 2021,” arXiv preprint arXiv:2107.08430, 2021.
- “Learning to predict where humans look,” in Proc. IEEE Int’l Conf. Computer Vision, 2009, pp. 2106–2113.
- Léon Bottou, “Large-scale machine learning with stochastic gradient descent,” in Proc. Int’l Conf. Computational Statistics, 2010, pp. 177–186.
- “Salicon: Saliency in context,” in Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, 2015, pp. 1072–1080.
- “User-centric visual attention estimation based on relationship between image and eye gaze data,” in Proc. IEEE Global Conf. Consumer Electronics, 2018, pp. 44–45.
- “User-specific visual attention estimation based on visual similarity and spatial information in images,” in Proc. IEEE Int’l Conf. Consumer Electronics-Taiwan, 2019, pp. 479–480.
- “Few-shot personalized saliency prediction with similarity of gaze tendency using object-based structural information,” in Proc. IEEE Int’l Conf. Image Processing, 2022, pp. 3823–3827.
- “What do different evaluation metrics tell us about saliency models?,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 41, no. 3, pp. 740–757, 2018.