Connecting the Dots: Surfacing Structure in Documents through AI-Generated Cross-Modal Links

Abstract: Understanding information-dense documents like recipes and scientific papers requires readers to find, interpret, and connect details scattered across text, figures, tables, and other visual elements. These documents are often long and filled with specialized terminology, hindering the ability to locate relevant information or piece together related ideas. Existing tools offer limited support for synthesizing information across media types. As a result, understanding complex material remains cognitively demanding. This paper presents a framework for fine-grained integration of information in complex documents. We instantiate the framework in an augmented reading interface, which populates a scientific paper with clickable points on figures, interactive highlights in the body text, and a persistent reference panel for accessing consolidated details without manual scrolling. In a controlled between-subjects study, we find that participants who read the paper with our tool achieved significantly higher scores on a reading quiz without evidence of increased time to completion or cognitive load. Fine-grained integration provides a systematic way of revealing relationships within a document, supporting engagement with complex, information-dense materials.

• The paper introduces an entity-and-link framework that uses GPT models and Molmo to connect fine-grained concepts across scientific paper text, figures, captions, and appendices.
• The augmented reading interface significantly improved quiz response quality (p = 0.0000294), especially when related information was two to four paragraphs apart, while showing no significant increase in completion time or NASA TLX workload scores.
• The study found that figure points and persistent reference panels were the most useful features, but benefits at extreme distances remain uncertain and depend on document type, reader expertise, model quality, and limited evaluation samples.

Overview

This paper addresses the problem of information fragmentation in long, multimodal documents such as scientific papers, where details relevant to a single concept are scattered across body text, figures, captions, tables, and appendices. The authors propose a framework for fine-grained integration, built on two primitive units: entities (discrete, semantically meaningful items such as claims, definitions, or figure elements) and links (relationships between entities, such as explaining, defining, or exemplifying). The framework is instantiated as an augmented reading interface for research papers, implemented as an extension of the ACM Digital Library, and evaluated through a formative study, a think-aloud study, and a controlled between-subjects experiment. The central empirical claim is that fine-grained cross-modal links significantly improve reading comprehension without increasing completion time or self-reported cognitive load (2602.16895).

Reading challenges from the formative study

A think-aloud formative study with ten researchers (nine PhD students, one postdoc) across disciplines including experimental physics, operations, computer science, anthropology, and history of science identified three recurring challenges:

• Fragmentation: figures were essential but hard to connect with related prose, especially when placed far from their references, split across distant sections, or relegated to appendices.
• Complexity: 8 of 10 participants cited dense figures with unclear notation, ambiguous labels, confusing color schemes, or small intricate details as major obstacles.
• Interpretation: readers struggled to confirm that their reading of a figure matched authorial intent — for example, which side of a graph represented improvement.

These challenges motivated a design target of surfacing relationships among small distributed details rather than operating at section- or document-level granularity, which characterizes most prior summarization and annotation tools.

Framework and system instantiation

The framework's abstraction — entities plus typed links — is deliberately minimalist so it can manifest across modalities (text, image, video, audio, AR/VR). The paper's instantiation targets figure-heavy scientific papers. Entities are extracted by a pipeline combining OpenAI GPT models (gpt-4o-2024-08-06 in early versions; gpt-5-2025-08-07 in the final evaluation) with Molmo (allenai/Molmo-7B-D-0924) for grounding visual entities to image coordinates. Text processing identifies phrases referring to those entities in captions and body passages constrained to explicit figure mentions (located via regular expressions), unifying paraphrastic references such as "Step 1" and "the first step." A separate pass generates standalone entity descriptions and retrieves related passages from throughout the paper.

The interface exposes these entities and links through four affordances:

• Consolidation: a persistent reference panel showing a zoomable figure copy, the entity description, direct references, and other related passages.
• Navigation: clickable passage links that scroll to their source locations, plus a visual index of figure thumbnails acting as a graphical table of contents.
• Exploration: interactive "figure points" and highlighted phrases with bidirectional hover activation.
• Decomposition: "figure scans" that step through one entity at a time with descriptions beneath the caption.

Iterative design yielded several instructive lessons: avoid visual overload by pruning points without textual references; supplement rather than replace original captions; embed visual content rather than floating it; and distinguish "direct references" from "other related passages." Notably, the authors concede that originally designed comprehensive "walkthroughs" had to be downgraded to lightweight "figure scans" because model performance at the time was insufficient — a candid acknowledgment that framework ambitions were bounded by technical capacity.

Design validation study

A think-aloud study with 14 HCI-affiliated participants examined how the affordances were actually used. Figure points were widely praised for sustaining attention and supporting targeted search; the reference panel was used by most participants (8/14) for side-by-side cross-referencing. Half of participants accessed AI-generated descriptions, but reception was mixed: some found them explanatory, while others judged them overly general or repetitive — feedback that directly motivated regeneration with GPT-5 before the comparative study. An unanticipated use emerged: some participants used passage links to verify AI descriptions, treating the augmentations as objects of scrutiny rather than trusted answers.

Comparative study results

The between-subjects study recruited 18 senior engineering undergraduates (9 per condition after excluding 38 pilot participants), who read one HCI stimulus paper under either the augmented or baseline interface and answered a ten-question open-book quiz scored 0–2 per question by two external annotators (Krippendorff's $\alpha = 0.75$).

Response quality improved significantly in the experimental condition (Mann–Whitney $U = 5042.5$, $p = 0.0000294$, $r = -0.33$), with a clear shift toward fully correct responses. Per-question gains were significant for questions 6, 8, and 9 initially, with questions 6 ($r = -0.852$) and 9 surviving Bonferroni correction.

The paper's most interesting analysis groups questions by the paragraph-distance between details needed for a correct answer. Improvement was significant at distances of two paragraphs ($U = 223.5$, $p = 0.00115$) and four paragraphs ($U = 75$, $p = 0.00079$, surviving Bonferroni), but not at far distances: question 7 required locating an appendix figure 62 paragraphs away, showed a slight negative trend, and only one participant (experimental) earned full credit — by noticing the critical detail inside a generated entity description. The authors state plainly that the benefit may drop off at very large distances and that participants may not expect critical details to live in appendix images; they also note the statistical caveat that significance was observed only at two- and four-paragraph distances.

Two null results carry equal weight. Despite adding roughly seven points per figure and 116 highlighted phrases, the augmented interface produced no significant difference in time to completion ($p = 0.320$) or on any of the six NASA TLX dimensions (all $U = 5042.5$0, all $U = 5042.5$1). Importantly, TOST equivalence tests did not establish statistical equivalence for time or cognitive load — the honest interpretation is "no evidence of harm," not demonstrated parity. The authors hypothesize that faster detail retrieval may have been offset by extra verification of other augmentations.

Preference data reinforced the pattern: all eight responding experimental participants rated figure points and the reference panel as useful or very useful, while popout and zoom/pan figures received the lowest ratings. Debrief interviews showed baseline participants dominated by time pressure and scrolling (5/9 reported substantial scrolling), whereas experimental participants framed reading around interface use; summaries were the single most requested feature overall (10/18).

Limitations

The evaluation rests on narrow footing in several respects. All comparative-study data comes from a single HCI stimulus paper read once by demographically homogeneous upper-undergraduate participants with limited research experience, over a 25-minute session — leaving open whether effects generalize to other document types, expert readers, or sustained real-world reading. Sample sizes were small, limiting statistical power particularly for the null results on time and cognitive load, and equivalence was never established. The instantiation depends on specific commercial and open-source models (GPT-4o, GPT-5, Molmo); results could vary with different underlying models, and the corpus was restricted to HTML papers from the ACM Digital Library, mostly in HCI and NLP. Finally, approximately five of ninety-plus AI-generated descriptions were manually revised for the final study, meaning the evaluated artifact was not purely machine-generated.

Conclusion

The paper contributes a generalizable entity-and-link framework for fine-grained document integration, grounded empirically at every stage: formative identification of fragmentation/complexity/interpretation challenges, iterative design shaped by usability failures, and a controlled study demonstrating significantly higher quiz quality at no measurable cost in time or perceived load. The distance-grouped analysis adds nuance — cross-modal links help most when related details are moderately separated (two to four paragraphs), with benefit uncertain beyond that range. The work positions complex documents as networks of entities and links and leaves open how such networks perform at extreme distances, across domains beyond computer science, and under different generative backends.