Learning Visual Importance for Graphic Designs and Data Visualizations (1708.02660v1)

Abstract: Knowing where people look and click on visual designs can provide clues about how the designs are perceived, and where the most important or relevant content lies. The most important content of a visual design can be used for effective summarization or to facilitate retrieval from a database. We present automated models that predict the relative importance of different elements in data visualizations and graphic designs. Our models are neural networks trained on human clicks and importance annotations on hundreds of designs. We collected a new dataset of crowdsourced importance, and analyzed the predictions of our models with respect to ground truth importance and human eye movements. We demonstrate how such predictions of importance can be used for automatic design retargeting and thumbnailing. User studies with hundreds of MTurk participants validate that, with limited post-processing, our importance-driven applications are on par with, or outperform, current state-of-the-art methods, including natural image saliency. We also provide a demonstration of how our importance predictions can be built into interactive design tools to offer immediate feedback during the design process.

• The paper introduces a novel neural network approach that learns pixel-wise visual importance using crowdsourced BubbleView data, validated against human perception.
• It employs fully convolutional networks optimized with cross-entropy loss, outperforming traditional saliency methods in predicting design element importance.
• The model's accurate predictions facilitate practical applications such as design retargeting, thumbnailing, and interactive feedback within automated design tools.

Overview of "Learning Visual Importance"

The paper "Learning Visual Importance" introduces a novel approach to understanding and predicting the visual importance of elements within graphic designs and data visualizations. The focus is on developing automated models using neural networks trained on crowdsourced human interactions, specifically clicks and importance annotations. This methodology leverages large-scale datasets to enhance prediction accuracy and to validate the models against human perception metrics such as eye movements and ground truth importance.

Data Collection and Methodology

The authors collected data using the BubbleView interface to approximate attention patterns as an alternative to eye tracking, which can be costly and impractical for large datasets. They gathered BubbleView data for over 1,400 data visualizations and used graphic design importance annotations from a prior paper. The datasets provided diverse sources and formats essential for training convolutional neural networks (CNNs). By comparing BubbleView clicks to both eye movements and annotation data, the authors demonstrated that these clicks effectively capture human attention's key aspects.

Neural Network Models

The paper employs fully convolutional networks (FCNs) inspired by recent advances in computer vision, targeting saliency modeling with a specific focus on graphic designs and data visualizations. The networks are trained using cross-entropy loss optimized for real-valued importance predictions, differing from traditional saliency approaches focused on discrete classifications. This innovative application of FCNs aims to predict pixel-wise importance without manual annotations, allowing for broader applicability across various design formats.

Evaluation and Results

The authors conducted extensive evaluations using two main metrics: Kullback-Leibler divergence and cross-correlation. These metrics assess the models' prediction alignment with human importance annotations and eye fixations. Results show that the FCN model outperforms existing natural image saliency methods when applied to BubbleView click predictions and provides competitive accuracy with respect to predicting eye fixations. Their analysis includes ranking importance of graphical elements based on their predicted and ground truth values, emphasizing successful localization of titles and prominent textual elements by the models.

Additionally, for graphic designs, the introduced model demonstrates improvements over previous methods in automatic importance prediction, confirmed through root-mean-square error (RMSE) and R² evaluations. These results are pivotal given the model's efficiency and speed, with execution times suitable for interactive applications.

Implications and Applications

Predictions from the model form the basis for practical applications such as retargeting and thumbnailing, which are critical tasks in managing visual content for various display formats and enhancing search retrieval efficiency. These applications utilize predicted importance to preserve and highlight key content areas within designs, ensuring that visual integrity is maintained across different resolutions and formats.

Moreover, the work explores the potential for embedding importance prediction into interactive design tools, showcasing how these models can provide real-time feedback and assist in optimizing design layouts. Future developments in AI may focus on integrating importance prediction models with broader design and media contexts, enhancing design automation and user experience.

Conclusion

"Learning Visual Importance" contributes substantial advancements in automating the understanding of visual importance within graphic and data design contexts. The paper effectively illustrates how neural networks can bridge the gap between computational efficiency and human-centered design perceptions. Its methodology, results, and applications offer a compelling foundation for exploring AI-driven design tools, with promising implications for future research and development in computer vision and interactive systems.