LLM-for-X: Application-agnostic Integration of Large Language Models to Support Personal Writing Workflows (2407.21593v1)

Abstract: To enhance productivity and to streamline workflows, there is a growing trend to embed LLM functionality into applications, from browser-based web apps to native apps that run on personal computers. Here, we introduce LLM-for-X, a system-wide shortcut layer that seamlessly augments any application with LLM services through a lightweight popup dialog. Our native layer seamlessly connects front-end applications to popular LLM backends, such as ChatGPT and Gemini, using their uniform chat front-ends as the programming interface or their custom API calls. We demonstrate the benefits of LLM-for-X across a wide variety of applications, including Microsoft Office, VSCode, and Adobe Acrobat as well as popular web apps such as Overleaf. In our evaluation, we compared LLM-for-X with ChatGPT's web interface in a series of tasks, showing that our approach can provide users with quick, efficient, and easy-to-use LLM assistance without context switching to support writing and reading tasks that is agnostic of the specific application.

• The paper presents an application-agnostic interface that reduces editing task times by approximately 40% compared to traditional methods.
• It employs a combination of OS-level hooks, accessibility APIs, and browser extensions to facilitate seamless LLM query handling and real-time diff view previews.
• User studies revealed improved usability scores and lower effort ratings, demonstrating significant practical benefits for diverse writing tasks.

The paper introduces a system-wide interface that enables application-agnostic integration of LLMs into virtually any text-based software environment. The approach is realized through an OS-level background service that monitors for global keyboard shortcuts and facilitates interaction with LLM backends via a lightweight pop-up UI overlay. This design avoids the conventional copy–paste paradigm and minimizes context switching, thereby streamlining workflows across native applications and web apps alike.

The system architecture relies on several technical components:

• OS-Level Background Service:

Implemented in C# using .NET APIs, the service registers a global keyboard hook to detect shortcut triggers. It leverages the Windows UI Automation API (UIA) to extract selected text and additional contextual information from foreground applications. In cases where UIA is unavailable, a clipboard fallback mechanism is employed.

• Native and Web-Based Interfaces:

For native applications, the integration is achieved via accessibility APIs, where the service retrieves the current window’s properties (such as window title and Process ID) as contextual cues to augment the LLM prompt. For web applications, a dedicated browser extension uses DOM APIs to capture text selections and related content. Communication between the browser extension and the native service is handled via the Native Messaging API, ensuring a seamless user experience.

• LLM Query Handling:

The system supports both interaction via emulated chat interfaces (mimicking standard LLM web UIs like those of ChatGPT, Gemini, etc.) and direct API calls to the LLM backends. In the emulated mode, the extension simulates user input and polls the DOM for the progressive appearance of the LLM’s response. The retrieved response is then previewed and can be inserted directly into the originating application through simulated keystrokes, preserving the contextual integrity of the text input (e.g., ensuring that the action is reflected in the application’s native undo/redo stack).

• Interaction Design and Features:
  • Predefined Commands and Custom Query Input: Users can trigger standard actions (e.g., “fix spelling mistakes”, “explain”, “translate”) with numeric shortcuts, or type any specific query.
  • Diff View Previews and Iterative Refinement: For editing tasks, the system presents a side-by-side diff view that highlights changes, enabling users to refine prompts iteratively while maintaining conversational context.
  • Direct Insertion Versus Replacement: Depending on modifier keys (such as SHIFT), the system distinguishes between replacing selected text or appending the LLM response below the current selection.
  • Contextual Augmentation: The prompt sent to the LLM is padded with additional context that includes the application name and window title, along with surrounding text if configured by the user. This enhances the LLM’s capacity to generate contextually relevant responses.

The evaluation comprises a controlled user paper with 14 participants who performed writing, reading, and coding tasks using both the proposed system and the standard ChatGPT web interface. Key quantitative findings include:

• Editing Task Performance:

A statistically significant reduction in task completion times was observed during editing tasks, with an average of 31.71 seconds using this system versus 51.14 seconds via ChatGPT (p < 0.05). This constitutes an approximate 40% speed improvement.

• Usability and Effort Metrics:

The system’s usability was rated significantly higher on the System Usability Scale (SUS), with average scores of 62.54 compared to 51.68 for ChatGPT. Additionally, participants reported lower effort scores on the NASA Task Load Index (TLX), particularly in terms of ease of use.

• Qualitative Feedback:

Users appreciated the elimination of context switching and the efficiency of keyboard shortcuts. Some users, however, noted that the familiar conversational style of ChatGPT provided a friendlier experience, indicating room for integrating more personalized features without compromising efficiency.

The paper also details the implementation nuances, such as simulating keypress events (using functions like SendKeys.SendWait) to ensure that LLM responses integrate natively into the target application’s editing flow. Moreover, the system is designed to recognize when a target element is non-editable and adapts its UI (by, for instance, hiding the TAB button) accordingly.

In summary, the paper demonstrates that the proposed system can significantly enhance text manipulation tasks by providing efficient in-situ LLM assistance across diverse applications. The technical contributions of integrating accessibility APIs, browser extensions, and LLM backends via both emulated interactions and direct API calls together form a robust framework for extending LLM services without the need for application-specific subscriptions. The work lays a solid foundation for further enhancements, such as incorporating multimodal context or personalized interaction cues, to further streamline user workflows in varied computing environments.