Visual Editing with LLM-based Tool Chaining: An Efficient Distillation Approach for Real-Time Applications (2410.02952v3)

Abstract: We present a practical distillation approach to fine-tune LLMs for invoking tools in real-time applications. We focus on visual editing tasks; specifically, we modify images and videos by interpreting user stylistic requests, specified in natural language ("golden hour"), using an LLM to select the appropriate tools and their parameters to achieve the desired visual effect. We found that proprietary LLMs such as GPT-3.5-Turbo show potential in this task, but their high cost and latency make them unsuitable for real-time applications. In our approach, we fine-tune a (smaller) student LLM with guidance from a (larger) teacher LLM and behavioral signals. We introduce offline metrics to evaluate student LLMs. Both online and offline experiments show that our student models manage to match the performance of our teacher model (GPT-3.5-Turbo), significantly reducing costs and latency. Lastly, we show that fine-tuning was improved by 25% in low-data regimes using augmentation.

Overview of "Visual Editing with LLM-based Tool Chaining: An Efficient Distillation Approach for Real-Time Applications"

The paper presents a methodology that leverages LLMs for real-time visual editing tasks, specifically focusing on modifying images and videos based on user input defined in natural language. This research introduces an approach to distill a smaller, open-source student LLM from a larger, proprietary teacher LLM, such as GPT-3.5-Turbo, optimizing it for cost and latency constraints inherent in production settings. The practical significance of this work lies in enabling advanced visual editing capabilities in applications with real-time performance requirements.

Methodology

The core of the method involves a process of fine-tuning where a smaller student LLM learns from a teacher LLM via tool chaining and behavioral signals. The process can be broken down into several components:

1. Data Collection: The authors compiled a dataset containing unique user intents and the corresponding actions taken by the teacher LLM in fulfilling these intents. The intent here is to focus on high-quality data as determined by user behavioral signals, such as export frequency.
2. Distillation Framework: A combination of auto-regressive models and sequence-to-sequence models were fine-tuned using the collected dataset. This involved tuning open-source models like Llama-2-7b-chat-hf and FlanT5-base, aiming to replicate the teacher LLM’s performance while reducing latency and cost.
3. Evaluation Metrics: The paper outlines both offline and online evaluation strategies. Offline metrics include tool-selection and quality scores, measuring how well the student LLM can predict tool parameters compared to the teacher. Online evaluations involved A/B testing to benchmark user satisfaction and engagement.
4. Data Augmentation: To address low-data scenarios, a novel augmentation strategy was employed. This involved generating analogous user intents using another LLM, thereby enhancing the training dataset incrementally.

Key Findings

• The student LLMs, specifically FlanT5-base, achieved competitive performance relative to the teacher model, achieving similar levels of relevance and quality in visual outputs.
• The augmentation approach led to a 25% improvement in low-data regimes, showcasing the effectiveness of using LLMs for data generation in resource-constrained environments.
• The practical deployment of the models demonstrated significant reductions in latency (1.38s on a less expensive GPU for FlanT5-base) compared to the teacher LLM, affirming the feasibility of deploying these models in a production setting without compromising the user experience.

Implications and Future Work

This research contributes to the growing field of employing LLMs for multimodal tasks, particularly in real-time mobile applications. The proposed distillation approach paves the way for democratizing advanced editing capabilities through smaller, accessible models, opening avenues for industry applications where computational resources are limited.

Future research directions suggested include:

• Integrating rationale as supplementary supervision for further improving model fine-tuning.
• Exploring models' performance across different languages and extending the methodology to other editing features.

The openness of the code and dataset invites further collaboration and development, encouraging the adaptation of the demonstrated approaches to an expanded set of visual editing tasks and tools. This research highlights the potential of LLMs to interface with specialized applications, driving innovation in real-time AI-driven content creation.