VIS-ReAct: LLM Framework for Sensemaking Refinement

• VIS-ReAct is a two-agent LLM framework that refines multi-document sensemaking reports by mapping visual semantic interactions to targeted edit plans.
• The system distinguishes itself by separating reasoning and action, ensuring semantic fidelity and precise updates without unnecessary global paraphrasing.
• Empirical results show that VIS-ReAct balances high precision and recall compared to baselines, while exposing its inference process for greater transparency.

VIS-ReAct is a two-agent LLM framework for refining multi-document sensemaking reports by interpreting and acting upon users’ semantic interactions in a visual workspace. In this setting, semantic interactions—such as highlighting passages, annotating notes, and creating or reorganizing clusters—are treated as semantically meaningful operations that externalize analysts’ evolving hypotheses, evidence, and organizational rationale rather than as mere UI events. The framework maps a newly observed interaction set to an intent representation, a refinement plan, and an updated report, with the stated goals of targeted refinement, semantic fidelity, and transparent inference (Tang et al., 2 Oct 2025).

1. Motivation and problem setting

VIS-ReAct is motivated by a recurrent failure mode in LLM-assisted sensemaking report writing: conventional systems regenerate entire drafts whenever the workspace changes. Even with fixed temperatures, this behavior induces paraphrasing or alteration of unrelated content, which makes incremental change tracking difficult and weakens the analyst’s control over targeted refinements. The framework is designed for settings in which reports are refined iteratively as visual workspaces evolve through clusters, highlights, and notes (Tang et al., 2 Oct 2025).

The underlying claim is that naïvely presenting pairs of workspaces to an LLM for “diffing” is unreliable. Subtle differences may be missed, user intent may be misread, and context needed to propagate only relevant edits may be lost. In the formulation used by VIS-ReAct, contemporary LLM-based report generation lacks three components: an explicit model for the semantics of workspace operations, intent inference over sequences of interactions, and a mechanism to plan targeted edits that preserve semantic fidelity without overediting (Tang et al., 2 Oct 2025).

This framing situates VIS-ReAct within visual analytics and human-LLM collaboration. Semantic interactions are interpreted as intent signals embedded in the analyst’s workspace manipulations, and report refinement becomes a reasoning-and-editing problem over those signals rather than a pure summarization problem.

2. Formalization of workspaces, interactions, and objectives

The framework formalizes the workspace at step $t$ as $W_t$ and the corresponding report as $R_t$. A workspace comprises clusters, documents, and visual marks. Clusters are denoted $C_t=\{c_k\}$ with membership $M(c_k)\subset D$. Highlights are represented per document as weighted text spans, $H_t(d)=\{(\text{span}, w)\}$, where weights encode frequency of highlighting. Notes are free-text annotations attached either to clusters, $N_t(c_k)$, or to documents, $N_t(d)$ (Tang et al., 2 Oct 2025).

A semantic interaction is represented as

$$s_t=\langle \text{type}, \text{target}, \text{payload}, \tau\rangle,$$

where $$\text{type}\in\{\text{AddCluster}, \text{RemoveCluster}, \text{ReorganizeCluster}, \text{AddHighlight}, \text{EditHighlight}, \text{RemoveHighlight}, \text{AddNote}, \text{EditNote}, \text{RemoveNote}\}$$, $W_t$0, $W_t$1 carries identifiers, spans, weights, note text, membership changes, and related parameters, and $W_t$2 is a timestamp. The interaction sequence is obtained by diffing consecutive workspaces:

$W_t$3

The report-update objective is defined as

$W_t$4

where $W_t$5 is an intent representation and $W_t$6 is a refinement plan. The explicit goals are targeted refinement, semantic fidelity, and transparent inference. In the state representation used by the framework,

$W_t$7

and each cluster paragraph in $W_t$8 should align with $W_t$9; the report follows BLUF with a summary paragraph, paragraphs for clusters, and a conclusion (Tang et al., 2 Oct 2025).

The interaction ontology is deliberately simple. Cluster operations include AddCluster, RemoveCluster, and ReorganizeCluster; highlight operations include AddHighlight, EditHighlight, and RemoveHighlight; note operations include AddNote, EditNote, and RemoveNote. This bounded ontology supports programmatic extraction of changes and constrains the subsequent reasoning problem.

3. Two-agent architecture and end-to-end workflow

VIS-ReAct uses a two-agent architecture. The LLM Analysis Agent performs the “Reason” stage: it interprets $R_t$0 in the context of $R_t$1 and $R_t$2, infers user intent $R_t$3, and produces a structured refinement plan $R_t$4. The LLM Refinement Agent performs the “Act” stage: it executes $R_t$5 to update the prior report while avoiding global paraphrasing, keeping unaffected sections unchanged, and preserving the report’s BLUF structure (Tang et al., 2 Oct 2025).

The end-to-end pipeline contains four steps. First, the current workspace $R_t$6 is converted into structured text via ReSPIRE, including cluster and document structure, highlights with weights, and notes. Second, semantic interactions are extracted by programmatically diffing $R_t$7 and $R_t$8 to obtain $R_t$9, following the established interaction types for clusters and visual marks. Third, the analysis agent takes $C_t=\{c_k\}$0, $C_t=\{c_k\}$1, and $C_t=\{c_k\}$2 as input to infer user intent and synthesize a refinement plan. Fourth, the refinement agent applies that plan to $C_t=\{c_k\}$3, updating only targeted sections and preserving the report format (Tang et al., 2 Oct 2025).

The refinement mechanism is constrained rather than generative in the broad sense. Its operational rules are: no global paraphrasing; modify only the paragraphs indicated in the plan, namely summary, affected cluster paragraphs, and conclusion; preserve BLUF structure and formatting. A plausible implication is that VIS-ReAct treats report editing as a typed, locality-sensitive transformation problem instead of a document-level rewrite problem.

The paper also presents a procedural view of the loop through VIS_ReAct_Refine(W_prev, R_prev, W_curr), in which SerializeWorkspace, DiffWorkspace, LLM_Analysis, and LLM_Refinement are composed, and an optional transparency artifact is returned as a log containing inferred intent, the plan, and the change log. That formulation makes explicit that workspace diffing is externalized as a programmatic step rather than delegated to the LLM.

4. Reasoning, planning, and transparent inference

The analysis agent consumes $C_t=\{c_k\}$4, $C_t=\{c_k\}$5, and $C_t=\{c_k\}$6 and produces two intermediate outputs: an intent inference $C_t=\{c_k\}$7 and a refinement plan $C_t=\{c_k\}$8. The prompt design uses a structured scratchpad with five sections: a context digest summarizing $C_t=\{c_k\}$9 and salient parts of $M(c_k)\subset D$0; an interaction summary enumerating $M(c_k)\subset D$1 grouped by type and target; intent hypotheses for each interaction; evidence grounding that maps highlights, notes, and clusters to entities, citations, and locations; and a refinement plan specifying target paragraphs, edit types, content specifications, and constraints (Tang et al., 2 Oct 2025).

The refinement plan is typed at paragraph level. It specifies target paragraph(s) from $M(c_k)\subset D$2, edit type from $M(c_k)\subset D$3, and a content specification describing entities, relations, and citations to be mentioned. In the additional formalization, the plan is written as

$M(c_k)\subset D$4

Intent inference is further expressed as

$M(c_k)\subset D$5

with the analysis agent approximating this mapping through structured reasoning. The reasoning strategy explicitly includes chain-of-thought in the sense of articulating how each semantic interaction implies edits. Programmatic diffing is treated as tool-use: the LLM reasons over a supplied interaction set rather than attempting to compute workspace differences itself (Tang et al., 2 Oct 2025).

Transparent inference is a core property of the framework. It refers to exposing what the model inferred about user intent from $M(c_k)\subset D$6, how those inferences were grounded in workspace elements, and which planned edits were executed. VIS-ReAct presents an inference log and a timeline of condensed analysis outputs, enabling a reviewable chain from interaction to intent to plan to edits. This is intended to mitigate the opacity of conventional LLM rewriting and to support verification and trust calibration.

5. Evaluation protocol and empirical findings

The evaluation uses “The Sign of Crescent,” a dataset of 41 fictional intelligence reports detailing three terrorist plots across three U.S. cities. The reported workspaces consist of 35 pairs of original and modified workspaces, each with a 10-document plot. Interaction types cover 13 combinations across highlights, notes, and cluster reorganizations, including adding, removing, and modifying elements, plus control cases with no interactions. Reports follow BLUF: a summary paragraph, one paragraph per cluster, and a conclusion paragraph. The model used for both agents is gpt-4o-mini (Tang et al., 2 Oct 2025).

Two metric families are used. P1, Targeted Refinement at paragraph level, defines $M(c_k)\subset D$7 as correctly refined sections, $M(c_k)\subset D$8 as total refined sections, and $M(c_k)\subset D$9 as sections that should be refined, with

$H_t(d)=\{(\text{span}, w)\}$0

P2, Semantic Fidelity at sentence level, defines $H_t(d)=\{(\text{span}, w)\}$1 as the number of relevant edited sentences, $H_t(d)=\{(\text{span}, w)\}$2 as total edited sentences, $H_t(d)=\{(\text{span}, w)\}$3 as realized interactions, and $H_t(d)=\{(\text{span}, w)\}$4 as total interactions, with

$H_t(d)=\{(\text{span}, w)\}$5

Relevant content is determined from elements extracted from interactions: entities and citations for highlights, entities for notes, and names or locations for clusters (Tang et al., 2 Oct 2025).

The baselines are a direct summarization system that completely regenerates the report from the current workspace, as in ReSPIRE, and an ablation called VIS-ReAct without LLM analysis, which summarizes $H_t(d)=\{(\text{span}, w)\}$6 to partially refine the report without intent inference and detailed planning.

Method	P1 Targeted Refinement	P2 Semantic Fidelity
Baseline	0.752 / 1.000 / 0.858	0.348 / 0.694 / 0.463
Without analysis	0.975 / 0.652 / 0.782	0.582 / 0.526 / 0.553
VIS-ReAct	0.951 / 0.831 / 0.887	0.558 / 0.684 / 0.614

These results show that VIS-ReAct attains the best F1 on both P1 and P2, combining high precision with improved recall relative to the ablation, and better balancing relevant edits and interaction coverage than the direct-regeneration baseline. The baseline achieves perfect recall on targeted refinement because it rewrites broadly, but this comes with lower precision and markedly weaker semantic fidelity. The ablation yields the highest precision on P1, but at the cost of reduced recall, indicating under-refinement (Tang et al., 2 Oct 2025).

The case studies illustrate the operational role of intent inference. In one example, a user highlights a suspicious name in the cluster “Explosives Procurement” and adds a note linking “M” to “Hassan” regarding C-4 purchases. VIS-ReAct infers an intent to emphasize procurement links and clarify ambiguous note abbreviations; the resulting plan adds a sentence in the relevant cluster paragraph with named entities and citations, updates the summary to reflect heightened confidence, and may adjust the conclusion. In another example, a document is moved from “Logistics” to “Funding”; the system infers reattribution of evidence and correspondingly expands the “Funding” paragraph while trimming “Logistics.” A third pattern involves isolated interactions, such as single highlights or member removals, where the analysis agent supplies context by referencing related notes or cluster semantics, preventing superficial edits (Tang et al., 2 Oct 2025).

6. Position within related research and terminology

VIS-ReAct is explicitly framed as a ReAct-style system in the sense that it separates reasoning from action: the analysis agent reasons over structured interaction inputs to produce plans, and the refinement agent acts on those plans through targeted textual edits. The framework differs from typical tool-use settings because its “actions” are report edits grounded in a visual workspace ontology rather than arbitrary API calls. It is also distinguished from prior report-writing LLM systems that regenerate reports wholesale or paraphrase individual elements, since its design centers on semantic interaction awareness and a two-agent division of labor for partial, context-aware refinement (Tang et al., 2 Oct 2025).

The framework is also situated within human-in-the-loop visualization and sensemaking. Building on the ideas of Space to Think and semantic interaction, it operationalizes intent inference from workspace manipulations and makes that inferred intent machine-interpretable for refinement planning. This suggests a tighter coupling between visual analytic state and downstream narrative state than is typical in document-generation pipelines.

VIS-ReAct should be distinguished from REACT, “Recognize Every Action Everywhere All At Once,” which is a vision-language architecture for group activity recognition in video, with a Vision-Language Encoder block, Actor Fusion Block, and Action Decoder Block for text-conditioned actor localization and group-level prediction (Chappa et al., 2023). Despite the similar naming, the two systems address different problems: VIS-ReAct concerns multi-document sensemaking report refinement, whereas REACT concerns video-based group activity recognition.

7. Implementation characteristics, limitations, and future directions

The reported implementation integrates ReSPIRE to serialize visual workspaces, including clusters, document structure, weighted highlights by frequency, and notes. Both the analysis and refinement agents use gpt-4o-mini. The analysis prompt requests interaction summarization, intent inference, evidence grounding through entities and citations, and a structured plan specifying target paragraphs and edit types. The refinement prompt enforces the constraints of no global paraphrasing, editing only targeted paragraphs, preserving BLUF, and adhering to format. Context management is separated across agents: the analysis agent ingests $H_t(d)=\{(\text{span}, w)\}$7, $H_t(d)=\{(\text{span}, w)\}$8, and $H_t(d)=\{(\text{span}, w)\}$9, while the refinement agent ingests $N_t(c_k)$0, $N_t(c_k)$1, and $N_t(c_k)$2. Interaction diffing is programmatic, and no training or fine-tuning is reported; the method is purely prompt-based. Performance is reported as around 20 seconds per refinement for a 10-document workspace, and case-study details with analysis and refinement examples are said to appear in supplemental material (Tang et al., 2 Oct 2025).

Several limitations and threats to validity are identified. Iterative refinements can accumulate visible edits and produce reports that are less concise than one-shot generation. Inferred intent may diverge from true user intent, especially when notes are sparse or ambiguous, such as abbreviations like “M” or “C” without disambiguation. Noisy interactions can induce plausible but incorrect refinements, implying a need for verification mechanisms. Scalability is a concern because long interaction sequences and large workspaces strain context windows and complicate precise targeting. The approach also relies on programmatic extraction of a subset of interaction types because LLMs struggle to compute workspace diffs directly (Tang et al., 2 Oct 2025).

The proposed future directions follow directly from those limitations. They include multi-modal grounding that incorporates visual features such as layout, proximity, and linking gestures; formal user intent models that learn probabilistic mappings $N_t(c_k)$3 and prioritize edits via utility functions; automated conflict resolution across notes, highlights, and clusters; RAG integration to improve efficiency and grounding; personalization to model user strategies and preferences; and richer collaboration transparency through interactive timelines for review, auditing, and trust calibration. A plausible implication is that VIS-ReAct can be understood as an initial formalization of report refinement as sequential intent-grounded editing over visual analytic state rather than as a terminal solution to incremental sensemaking assistance.