VisPile: Visual Analytics for Text Analysis

• VisPile is a visual analytics system for multi-document text analysis that integrates LLMs and knowledge graphs to support intelligence analysts.
• The system combines interactive visual analytics, LLM-powered synthesis, and KG-based fact validation to streamline evidence organization and relationship discovery.
• Its user-driven workflow and provenance tracking mechanisms enhance traceability and facilitate iterative hypothesis building in practice.

VisPile is a visual analytics system for multi-document text analysis that integrates LLMs and knowledge graphs (KGs) within a human-in-the-loop sensemaking workflow, designed primarily for intelligence analysts faced with curated but unmanageably large text corpora. Combining interactive visual analytics, LLM-powered synthesis, and KG-based fact grounding, VisPile enables rapid evidence organization, relationship discovery, and validation of machine-generated claims. The system was developed in active collaboration with the U.S. Intelligence Community and is specifically architected to address the practical and epistemic constraints of real-world analytic sensemaking (Coscia et al., 10 Oct 2025).

1. Motivation and Sensemaking Problem Statement

The core motivation for VisPile is rooted in long-standing challenges facing intelligence analysts: the need to synthesize situational awareness and actionable hypotheses from large, fixed corpora of unstructured text. Existing workflows are hampered by slow manual review, difficulty in identifying relevant material when appropriate keywords are unknown, challenges in grouping and summarizing evidence, and the persistent risk of analytic conclusions not being sufficiently traceable to source documents.

Traditional visual text analysis systems provide partial solutions, such as topic modeling or interactive document layouts, but have not generally integrated recent AI approaches—particularly the synthesis and entity extraction capabilities of LLMs or the structured relational evidence provided by KGs—across the full iterative workflow of document grouping, pile-centric analysis, and machine output validation. VisPile addresses this by explicitly targeting the full sensemaking progression: from raw documents, through manual and automated grouping (piling), to fact and relationship extraction, and ultimately to rapid, evidence-supported insight reporting.

2. VisPile System Architecture and Data Flow

VisPile is a web application constructed using a Vue.js frontend. Its backend processes and indexes a fixed corpus of plain-text documents, providing metadata logging (title, date, length, topic label), topic assignments via BERTopic (“an LLM-based topic modeling approach”), entity and relationship extraction for KG construction, and embedding generation for semantic search.

The KG is built by feeding each document through GPT-3.5 Turbo to extract $<$entity, relationship, entity$>$ triples, which are stored as subject$\to$predicate$\to$object facts with provenance links to their source documents. Semantic deduplication is applied: duplicates are grouped by cosine similarity in embedding space, a representative fact is selected, and all documents associated with overlapping triples are aggregated. Within the interface, facts in the KG are ranked for topicality and “support” by embedding similarity to the pile’s text and by the number of supporting documents per fact, respectively.

The LLM back end is model-agnostic but, in the implementation described, is centered on GPT-3.5 Turbo. It is invoked for pile-level operations via carefully engineered prompts developed in collaboration with domain experts; these prompts are fully exposed in the interface for transparency. Semantic search and RAG-style retrieval use vector embeddings, e.g., OpenAI text-embedding-3.

The interface is structured around several coordinated views:

• Document list: supports keyword search, metadata filtering, BERTopic topic browsing, and embedding-based semantic retrieval.
• KG entity search: returns a ranked, citation-linked list of fact triples and enables pivot navigation across connected entities.
• Central “piles” workspace: analysts manually group candidate documents into piles, encode hypotheses in pile names, apply LLM/KG analysis, and refine piles iteratively.

3. The Pile Metaphor and Analyst-Driven Workflow

The “pile” is the canonical workflow abstraction, paralleling real-world intelligence-analytic practices of grouping document subsets by topic, hypothesis, or investigative line. Documents are dragged into spatial clusters (piles), renamed, sorted, duplicated, or refined as the analysis proceeds. Critically, all automated operations—LLM-based synthesis, KG-based fact retrieval—are scoped to the documents within a selected pile, ensuring analyst control over evidence aggregation and synthesis.

After pile creation, the analyst can select from a set of nine pre-engineered LLM prompts (with full customization support):

1. Analyze the documents for patterns and insights
2. Summarize the main events described
3. Extract relevant entities (people, locations, etc.)
4. Classify the relevant topics discussed
5. Generate potential questions that the documents raise
6. List analytic tasks to perform based on the documents
7. Explain concepts mentioned in the documents
8. Answer questions using the documents
9. Custom prompt with open-ended input

Prompts exhibit adjustable fields (question, entity type, targeted concepts) and temperature parameters; they are always displayed verbatim. Pile refinement is interactive: new evidence can be added or removed, prompts rerun, and hypotheses iteratively clarified.

4. LLM and KG Integration: Complementary Roles and Validation Mechanisms

VisPile is predicated on the complementary strengths and weaknesses of LLMs and KGs. LLMs offer rapid, open-ended synthesis—document summarization, event extraction, topic classification, question generation, and hypothesis formulation—thus enabling analysts to “read less and reason more.” However, LLMs are known to hallucinate, make omissions, or generate unsupported claims. KGs provide explicit, provenance-linked fact triples, supporting structured relationship traversal, contextualization, and evidentiary validation.

To facilitate robust sensemaking, VisPile supplies a set of validation tools:

• Extract: highlights entities surfaced by the LLM that match nodes in the KG; enables direct graph traversal starting from synthesized text.
• Link: links each LLM-generated sentence to its most semantically similar sentence among source documents (using sentence-level embedding similarity), color-coding and underlining sentences to expose provenance.
• Suggest: suggests additional documents from the full corpus based on semantic similarity to the LLM response, automatically augmenting the current pile with new evidence for corroboration or expansion.

These mechanisms enforce a mixed-initiative analytic mode: automation accelerates foraging and synthesis, but final evidence organization and reliability validation remain controlled by the analyst.

5. Interface, Visual Encoding, and User Interaction Paradigms

The system’s interface is intentionally text- and list-centric, avoiding cognitively burdensome node-link diagrams for large KGs. Document list items and pile blocks are the primary interaction points. The KG fact list shows up to five citation-linked fact triples per query, supporting entity-centric local exploration. LLM responses are plain text, with entities and relationships visually highlighted for linking and extraction. The spatial workspace allows direct manipulation of piles, and visual overlays on LLM outputs provide immediate cues to source-provenance mapping.

Analysts navigate between three main representational levels:

• Raw documents (always directly accessible)
• Intermediate structures (piles, LLM outputs, KG facts, entity highlights, provenance mappings)
• Synthesized insights (iteratively built hypotheses, validated relationships, and reportable findings)

6. Evaluation, User Feedback, and Observed Workflows

VisPile underwent an expert feedback study with six professional intelligence analysts experienced in visual text analytics. Each analyst spent 60 minutes working on the KRONOS dataset, following their own workflow, while feedback was recorded and analyzed thematically.

Analyst feedback emphasized:

• The value of reducing manual reading through LLM-generated overviews, summaries, and question prompts.
• The utility of the KG for targeted question answering and evidence discovery, particularly as a means to find source documents supporting specific facts.
• Strong preference for features that supported hypothesis iteration and allowed for staged analysis (starting broad, splitting piles over time, and validating LLM claims via source checking).
• Critical importance of traceability and skepticism toward LLM outputs; trust was constructed gradually and relied upon the availability of provenance overlays and evidence linking.

Common workflow patterns included initial broad pile formation followed by iterative refinement, and staged sequencing of LLM/KG operations—entity extraction and summary tasks for orientation, then deeper question answering and analytic task listing as understanding solidified.

7. Limitations, Design Goals, and Broader Context

The reported evaluation acknowledges limitations:

• The KRONOS corpus (845 short documents) is a proxy and may not reflect the complexity of real intelligence corpora.
• Only GPT-3.5 Turbo was tested for LLM tasks and KG extraction; the impact of alternative models or larger context windows is unstudied.
• No formal accuracy benchmarks, baseline comparisons, or timing analyses were conducted.
• Automated KG entity alignment (i.e., coreference resolution across mentions of people, places, events) is absent, likely limiting graph quality and fact matching.
• Lack of a node-link KG visualization may reduce analysts’ macroscopic understanding of graph structure, despite simplifying local evidence inspection.

Explicitly stated design goals (G1–G6) are: G1: Leverage LLM for open-ended document search. G2: Link KG data to source documents. G3: Run LLM sensemaking tasks within piles. G4: Traverse connected KG facts related to piles. G5: Enable validation operations on LLM responses. G6: Provide related context around KG facts.

The authors recommend further research on how generative AI affects analyst critical thinking and sensemaking, especially in task-diverse, trust-critical domains.

---

References:

• "VisPile: A Visual Analytics System for Analyzing Multiple Text Documents With LLMs and Knowledge Graphs" (Coscia et al., 10 Oct 2025)
• For the piling metaphor and design space: "A Generic Framework and Library for Exploration of Small Multiples through Interactive Piling" (Lekschas et al., 2020)