Contexty: Capturing and Organizing In-situ Thoughts for Context-Aware AI Support

Abstract: During complex knowledge work, people engage in iterative sensemaking: interpreting information, connecting ideas, and refining their understanding. Yet in current human-AI collaboration, these cognitive processes are difficult to share and organize for AI. They arise in situ and are rarely captured without interrupting the task, and even when expressed, remain scattered or reduced to system-generated summaries that fail to reflect users' cognitive processes. We address this challenge by enabling AI context that is grounded in users' cognitive traces and can be directly inspected and revised by the user. We first explore this through a probe system that supports in-situ snippet memoing, allowing users to easily share their cognitive moves. Our study (N=10) highlights the value of capturing such context and the challenge of organizing it once accumulated. We then present Contexty, which supports users in inspecting and refining these contexts to better reflect their understanding of the task. Our evaluation (N=12) showed that Contexty improved task awareness, thought structuring, and users' sense of authorship and control, with participants preferring snippet-grounded AI responses over non-grounded ones (78.1%). We discuss how capturing and organizing users' cognitive context enables AI as a context-aware collaborator while preserving user agency.

• The paper introduces Contexty, a system that enables users to externalize and structure in-situ cognitive context via snippet memoing, improving AI task alignment.
• It employs a canvas-based interface to organize captured snippets into hierarchical, inspectable memory cards that link cognitive artifacts.
• Empirical evaluation shows significant improvements in task awareness and thought structuring without increasing cognitive workload.

Contexty: A User-Inspectable Context System for Human-AI Collaboration

Motivation and Problem Formulation

Contemporary LLM-based assistants are constrained by limited, misaligned context representations. Most collaborative sensemaking occurs through moment-to-moment cognitive moves—interpreting, evaluating, and connecting information—yet existing LLM systems capture only what is explicitly prompted or inferred passively, omitting internal rationales, emergent criteria, and the “why” behind user actions. Furthermore, when such context is captured, it is usually fragmented, buried in chat logs, or reduced to abstract, system-centric summaries that are not user-inspectable.

The challenge thus decomposes into a capturing problem (fleeting in-situ thoughts are rarely externalized into the AI’s context) and a visibility problem (users cannot directly inspect or refine how their cognitive context is represented to the AI). This work interrogates and proposes direct affordances for externalizing cognitive traces and structuring them as user-inspectable, correctable AI context.

Snippet Memoing as Cognitive Context Capture

The authors developed a probe system supporting snippet memoing. This mechanism allows users, via global shortcuts, to instantly capture a screenshot or text snippet from any application, annotate it with a brief in-situ memo, and transmit the snippet-memo pair to the AI system.

Figure 1: The probe system for snippet memoing; (A) in-situ capture with memo UI, (B) accumulation and AI interpretation of snippet–memo pairs, (C) chat with AI grounded in accumulated context.

This design significantly reduces the articulation and interaction cost compared to multi-turn prompting, enabling users to externalize ongoing interpretations, rationales, and criteria as they naturally encounter them. Study data demonstrate that snippet memoing elicits not only what users are attending to, but why it matters, and anticipatory context about future task states. Notably, it lowers externalization friction, increases the specificity of user-authored context, and serves as a scaffold to anchor later reflective navigation.

However, as snippets accumulate, organizing them becomes a challenge. Participants struggled with connecting, grouping, and tracking the relationships or evolving criteria represented in their accumulated cognitive context.

Contexty: Inspectable and Correctable AI Context

To address the dual challenge of capture and organization, Contexty is introduced as a full system. It provides: (a) first-class integration of snippet-memoed content and other behavioral observations into AI context; (b) a canvas-based, user-inspectable context memory structure; and (c) direct manipulation and reorganization affordances to maintain context-task alignment.

Figure 2: Contexty’s main window, showing timeline and hierarchical overview of captured context, AI-generated context summary, composable and interactive canvas, and direct links to chat with contextually augmented AI.

Each discrete cognitive artifact (snippet, observation, chat) is rendered as a “memory card” that combines AI analysis—titles, tags, context interpretation—with full provenance and the user’s attached rationale.

Figure 3: A memory card, including AI-generated title, full provenance, captured content, tags, context interpretation, and the original user’s in-situ memo as a sticky note.

Cards are organized via an LLM-supported, hierarchical memory structure with named semantic branches and cross-item links. New memories are auto-grouped using LLM-based semantic similarity, and both manual and AI-assisted reorganization are supported via drag-and-drop and natural language commands. The structure remains fully correctable and inspectable by the user.

A floating widget ensures that context capture and inspection are available at any point in a user’s workflow.

Figure 4: The floating widget for quick snippet capture, observation toggling, and mirrored chat access.

For AI interaction, retrieved context is dynamically constructed by merging explicit user references (@-mentions, right-click->”add to chat”) and AI-selected, semantically relevant items (weighed by lexical overlap, tags, recency; with snippets prioritized).

Empirical Evaluation

A within-subjects study (N=12) compared Contexty and a baseline lacking the canvas but still leveraging snippet memoing. Quantitative and qualitative metrics assessed (1) task awareness, (2) thought structuring, (3) perceived AI understanding, (4) authorship, (5) controllability, and (6) overall satisfaction.

Figure 5: Post-condition survey results (7-point Likert); Contexty yields significant improvements in awareness, structuring, authorship, controllability, and satisfaction.

Contexty produced sizable improvements (all $p<0.05$), especially in task awareness ($\Delta=+1.33$, $p=.0039$) and thought structuring ($\Delta=+1.5$, $p=.0039$). Users reported that the canvas made their cognitive context visible, easier to revisit, and modifiable, promoting both active organization (editing, grouping) and passive sensemaking (overview, traceability).

Controllability increased without a corresponding rise in cognitive workload (NASA-TLX), indicating that curation costs were offset by regaining agency over context structuring.

Figure 6: Comparison of NASA-TLX workload scores for Contexty and Baseline; no significant increase in mental or temporal demands.

Moreover, an in-situ response preference probe examined the impact of snippet-grounded context on AI response utility. For queries resulting in substantively different answers (N=103 judgements), users selected snippet-informed responses over context-without-snippet 78.1% of the time.

Figure 7: Query funnel per participant: 48% of queries led to distinguishable context, of which snippet-augmented responses were preferred in 78% of cases.

Similarity analysis demonstrated robust divergence between response pairs, and only high NCD ($>$0.7) pairs were surfaced for preference rating to reduce noise. While snippet-augmented context typically conveyed user intent more effectively and increased relevance, a minority of responses without snippets were preferred for breadth or diversity, indicating that heavy context grounding can sometimes reduce creative latitude.

Theoretical and Practical Implications

Contexty demonstrates that direct, in-situ cognitive externalization (via snippet memoing) combined with user-inspectable and correctable context representations yields measurable improvements in human–AI task alignment, sensemaking support, and user agency. The results robustly support grounding AI context in user-authored signals and rendering that context visible, correctable, and reusable.

This architecture foregrounds critical challenges for long-term memory management in LLM agents: (1) balancing comprehensive passive observation (increasing recall, but introducing noise and curation costs) versus intentional user-managed artifacts, and (2) ensuring bi-directional translation between human-intelligible, spatial or hierarchical context views and AI-oriented sequence or vector representations.

Contexty’s findings align with, but extend, recent work on agentic memory for LLM agents [xu2025Amem], proactive user modeling [shaikh2025GUM], and canvas-based thought externalization [li2026orality], by demonstrating that the highest utility arises when user-initiated artifacts are directly inspectable in the AI’s context pipeline.

Limitations and Directions for Future Work

Experimentation was limited to single-session sensemaking and information foraging tasks. Generalization to longitudinal workflows or to content generation domains (writing, programming) is untested. The organizational structure implemented—a hierarchical memory + links model—may not be optimal for all tasks or for dynamic, long-horizon projects. Further, bridging user-centered spatial organizations with AI-centric structured memory remains an unsolved technical challenge.

Future work should investigate memory summarization, curation, and selective resurfacing techniques for long-term cross-session continuity, as well as mechanisms for aligning the evolving representational needs of users and AI systems. The risk of AI-generated structures steering user cognition, and the trade-off between agency and automation, remain salient open questions.

Conclusion

Contexty empirically validates that grounding AI systems in user-authored, in-situ cognitive context—and making that context directly inspectable and correctable—substantially improves alignment, task awareness, and sensemaking in human–AI collaboration. Integrating lightweight memoing with a user-aligned, manipulable canvas offers a template for future proactive, context-rich LLM systems. Moving beyond opaque, system-managed context to shared, human-intelligible representations is crucial to advancing transparent and user-controlled AI agents.