CogChat: Knowledge Graph-Augmented Conversational AI with Heterogeneous Graph Transformer for Cognitive Grounding in Design Generation

Abstract: LLM-based chat systems have become valuable tools for design practice, enabling rapid ideation and flexible task support. Yet these systems process designer utterances as generic sequences, maintaining context through recency rather than through any model of how the speaker organizes knowledge. In design conversation, this gap compounds as relational context decays between turns, identical words go unresolved across designers, and the conversation loops or restarts rather than deepens. We present CogChat, a real-time chat framework that grounds conversational AI in a personal heterogeneous knowledge graph constructed from each designer's input. The system extracts typed entities and relations into a heterogeneous graph, then applies a HGT (Heterogeneous Graph Transformer) to select structurally relevant nodes for response generation and to generate both intentional and exploratory probing questions. Technical evaluation shows that HGT-based entity selection outperforms both ungrounded LLM interaction and naive KG augmentation, which introduces noise that degrades response quality. A within-subjects study with nine professional designers indicates that grounding conversation in a relationally structured, designer-specific semantic context improves context retention, personalized intent interpretation, and conversational depth while reducing cognitive load. These findings suggest that structuring a designer's expressed concepts and relations as a dynamic knowledge graph can preserve relational context that fades across turns, pointing toward a graph-grounded approach to long-term context management in LLM-based interaction.

• The paper introduces CogChat, a GPT-based design assistant that incrementally builds a personal heterogeneous knowledge graph and uses a 3-layer, 8-head HGT to selectively ground responses in relevant concepts, materials, properties, and actions.
• The paper reports that KG+HGT outperformed both an LLM-only baseline and full graph injection, reaching 94.2 on RewardBench Reasoning, 62.3 on LongMemEval, and 88.7% on LoCoMo accuracy.
• The paper shows that professional designers completed tasks in 5.1 turns versus 9.4 for the baseline, reduced session dislikes from 3.67 to 0.22, and achieved faster, more personalized interactions, although evidence is limited by a nine-person study and short-term tasks.

Motivation and problem statement

CogChat addresses a structural limitation of LLM-based conversational systems in design practice: context is maintained through recency rather than through any model of how the speaker organizes knowledge. The authors identify three compounding failures that follow from this limitation—cross-turn decay of relational context (e.g., forgetting that "warm" referred to tactile grain rather than color temperature), failure to resolve designer-specific polysemy ("heavy" as visual density versus information overload), and conversational looping that forces designers to re-explain established ground. They argue these are symptoms of a single absence: no persistent, structured representation of the designer's expressed conceptual space. Notably, the system deliberately models only what designers explicitly express, not their latent cognition—a scope restriction stated clearly and maintained throughout.

The paper's central claim is that the bottleneck in LLM-based design conversation is not memory capacity but context curation. This claim is supported by an ablation design in which the Baseline condition receives the full conversation history within the model's context window yet still underperforms, isolating structure—not information availability—as the operative variable.

System architecture

CogChat grounds a GPT-based chat interface in a personal heterogeneous knowledge graph constructed incrementally from each designer's utterances. The pipeline runs five stages per turn:

1. Entity and relation extraction via GraphRAG-style prompting over GPT-4o, with entities typed against a lightweight design ontology (CONCEPT, DESIGN_ELEMENT, MATERIAL, PROPERTY, ACTION). Nodes are initialized with 512-dimensional CLIP embeddings; cross-turn answered_by edges link question entities to answer entities.
2. Link set creation with hard negatives drawn from structurally proximate but unconnected nodes (same-type unchosen entities, disliked entities, same-HDBSCAN-cluster nodes) and random negatives beyond the two-hop neighborhood. Edge weights adjust with explicit like/dislike feedback.
3. Subgraph sampling and HGT embedding using a 3-layer, 8-head Heterogeneous Graph Transformer with type-specific Q/K/V projections and relation-specific attention, producing 256-dimensional embeddings trained per-turn with BCE plus ranking loss. A cold-start bypass injects all nodes directly when the graph has fewer than five nodes.
4. Context-grounded generation: top-$k$ ($k \approx 20$) nodes ranked by cosine similarity to the message embedding are injected into the LLM prompt with type, relations, and neighborhood summaries. A continuity score blending node-set overlap and averaged-embedding similarity triggers a context warning when the conversation shifts direction.
5. Probing question generation partitioning normalized similarity scores into three zones: nodes above $\mu + \sigma$ seed intentional questions resolving underspecified concepts; the middle band seeds exploratory questions surfacing structurally adjacent but undiscussed territory; low-similarity nodes are excluded. Each question carries 2–3 suggested answers whose responses feed back into extraction.

Average turn latency is approximately 2.3 seconds on an RTX 4090, with latency stability attributed to HGT attending to sampled subgraphs rather than the full graph. The interface supports in-context knowledge lookup (double-clicking a term reveals its typed relations), which participants cited as increasing trust by making the system's accumulated understanding inspectable and correctable.

Technical evaluation

The authors benchmark three conditions—Baseline (LLM-only), KG-only (full extracted-graph injection), and KG+HGT (selective top-$k$ injection)—on four benchmarks using GPT-5.4 as base model across five runs at temperature 0. Representative results:

Benchmark	Metric	Baseline	KG-only	KG+HGT
ASQA	QA-Hit	78.0±1.9	81.8±1.8	83.2±1.7
RewardBench Chat Hard	—	68.4±2.4	71.2±2.0	91.6±2.2
RewardBench Reasoning	—	71.8±2.2	85.4±2.1	94.2±2.0
LongMemEval Overall	Contains Match	30.7±0.5	57.0±0.8	62.3±2.4
LoCoMo Overall	Accuracy	84.2±0.8	86.1±1.5	88.7±1.2

Two findings deserve emphasis. First, the largest gains appear on Chat Hard ($\Delta$23.2 over Baseline) and Reasoning ($\Delta$22.4), supporting the claim that benefits scale with relational complexity and ambiguity. Second, and more pointedly, KG-only frequently degrades relative to expectations—unfiltered graph injection introduces noise that competes for LLM attention, so more context can hurt when relevance is not enforced. This is the paper's strongest technical argument: selective heterogeneous grounding, not graph presence alone, drives quality. One caveat is that KGs for these benchmarks were constructed per-query from questions and reference passages using the same extraction pipeline, so the graphs are cleaner than those accumulated from noisy free-form dialogue; the authors partially address this concern with a reported extraction precision of 88.6% across 1,226 entities and 1,710 relations under strict LLM-based judgment.

User study

A within-subjects study with nine professional designers (mean experience 4.78 years, all active LLM users) compared the three conditions on counterbalanced design tasks, analyzed with Friedman tests, Wilcoxon signed-rank pairwise tests with Benjamini–Hochberg correction, and Kendall's $W$ effect sizes. Effect sizes were large throughout ($W > .6$ for most measures).

Context retention (RQ1). CogChat required 5.1 turns versus 6.0 (KG-only) and 9.4 (Baseline)—a 45.9% reduction versus Baseline ($p < .001$)—and completed tasks in essentially one session (1.0 vs. 2.1 for Baseline). Negative feedback collapsed from 3.67 dislikes/session (Baseline) to 0.22 (CogChat), with no significant difference between the two KG conditions, indicating that context loss rather than response quality drove dissatisfaction. Token-level IoU (.185 vs. .110 and .077) suggests HGT enables reuse of the designer's own vocabulary. A key control: KG-only and CogChat accumulated comparable graphs (~80–88 nodes, ~130 edges) and injected similar subgraph sizes per turn, but node utilization differed sharply (43.9% vs. 26.8%)—CogChat selects more precisely from a richer pool rather than consuming more context.

Personalized interpretation (RQ2). CUQ scores were 77.8 / 64.9 / 46.7 across CogChat / KG-only / Baseline, with the KG-only vs. CogChat gap itself significant. Probing Success Rate—the proportion of generated questions participants chose to answer—was .70 for CogChat versus .55 (KG-only) and .17 (Baseline); since skipping was free, this behavioral measure indicates graph-grounded questions were perceived as roughly four times more relevant than generic follow-ups. Interview evidence includes P8's report that "water properties" was correctly interpreted as gentle wave texture rather than transparency—an instance of designer-specific semantic resolution that generic processing missed.

Cognitive effort and depth (RQ3). NASA-TLX Mental Demand, Effort, and Frustration all followed Baseline > KG-only > CogChat, though the KG-only vs. CogChat difference was not significant here—the authors attribute load reduction primarily to context retention itself. Task completion time, however, did separate all pairs where it mattered: 14.1 min (CogChat) versus 17.9 (KG-only) and 19.5 (Baseline), implying that retention alone does not yield efficiency; personalized grounding converts retained context into faster outcomes. Eight of nine participants identified CogChat as producing the deepest conversation, citing specificity, multi-perspectival engagement, and progressive refinement. Agency/Ownership was rated highest for CogChat, with one participant distinguishing "the AI helped me" from "this feels like my design."

Limitations and open questions

The authors concede several constraints plainly. Extraction depends on GPT-4o's text-based capability and may fail on abstract or purely visual intents; multimodal knowledge graphs embedding visual elements directly remain unexplored. Cold-start performance is weak: before relational structure accumulates, CogChat's advantage over a strong LLM with clarifying questions and preference summaries is small. The study covers nine experts on short single-session tasks, so claims about long-term personalization are unsupported; longitudinal tracking of graph evolution, graph decay management (pruning outdated nodes, resolving contradictory constraints, handling intent pivots), blind expert ratings of final artifacts, and targeted ablations disentangling probing mechanisms from heterogeneous grounding are all explicitly left open. The authors also appropriately decline universal claims: benefits concentrate on relationally dense, ambiguity-heavy conversation, while KG-only often suffices for simple retention.

Conclusion

CogChat demonstrates that grounding LLM conversation in a designer-specific heterogeneous knowledge graph—with HGT-based selective entity injection and distribution-partitioned probing questions—improves context retention, personalized intent interpretation, conversational depth, and task efficiency relative to both ungrounded interaction and naïve full-graph augmentation. Its most consequential finding is comparative: structured selection, not additional memory, accounts for the gains, reframing long-term context management as a curation problem. The evidence rests on strong effect sizes in a small expert sample and benchmarks with synthetically constructed graphs, leaving longitudinal deployment, visual intent capture, and artifact-quality assessment as the principal unresolved questions.