Live Fitness Coaching as a Testbed for Situated Interaction (2407.08101v2)

Abstract: Vision-LLMs have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions, where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time, are an open challenge. In this work, we present the QEVD benchmark and dataset, which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching -- a task which intrinsically requires monitoring live user activity and providing immediate feedback. The benchmark requires vision-LLMs to recognize complex human actions, identify possible mistakes, and provide appropriate feedback in real-time. Our experiments reveal the limitations of existing state-of-the-art vision-LLMs for such asynchronous situated interactions. Motivated by this, we propose a simple end-to-end streaming baseline that can respond asynchronously to human actions with appropriate feedback at the appropriate time.

• The paper presents a novel dataset and benchmark comprising QEVD-FIT, capturing over 474 hours of annotated fitness video clips and live feedback.
• It introduces Stream-VLM, an end-to-end streaming vision-language model that uses action tokens to deliver timely, context-specific coaching.
• Experimental evaluations show that Stream-VLM outperforms existing methods in both fluency and temporal accuracy, highlighting its real-world potential.

Live Fitness Coaching as a Testbed for Situated Interaction

The research paper "Live Fitness Coaching as a Testbed for Situated Interaction" by Panchal et al. presents a novel dataset and benchmark aimed at exploring human-AI interaction in the domain of live fitness coaching. This domain is particularly challenging due to the need for continuous monitoring of user activity and providing timely, contextual feedback. The paper's contributions are manifold, ranging from the introduction of a comprehensive dataset to the proposal of a novel baseline model, Stream-VLM, tailored for this task.

QEVD Dataset and Benchmark

The Qualcomm Exercise Video Dataset (QEVD) is the cornerstone of this research. It comprises two primary subsets: QEVD-FIT-300K and QEVD-FIT-COACH.

QEVD-FIT-300K consists of over 474 hours of short video clips (approximately 5 seconds each), annotated with over one million question-answer pairs. These clips capture 148 different exercises with fine-grained variations, alongside 49 types of general activities. The annotations include fitness questions aimed at both high-level understanding and fine-grained details about the performance of the exercise.

QEVD-FIT-COACH comprises long-range videos (>3 minutes), annotated with live feedbacks. This subset forms the primary benchmark for assessing the performance of vision-LLMs in a real-world fitness coaching scenario. The dataset is unique in its inclusion of corrective feedback, which is crucial for developing assistive vision-LLMs capable of real-time interaction.

Methodology

The paper introduces Stream-VLM, a novel end-to-end streaming vision-LLM designed to address the limitations of existing models in providing asynchronous, situation-aware feedback. The architecture integrates a 3D Convolutional Neural Network (CNN) as the vision backbone, specialized for recognizing fine-grained human actions, with a LLaMA-2 LLM. Unique action tokens (<next> and <feedback>) are employed to enable the model to decide when to provide feedback, thus mimicking a real-time coaching scenario.

Experimental Evaluation

The authors conducted extensive evaluations using both zero-shot and fine-tuned models. State-of-the-art models such as InstructBLIP, Video-LLaVA, and Video-ChatGPT were evaluated in a simulated interactive scenario where they were prompted to provide feedback at regular intervals. The performance was assessed using metrics like METEOR, ROUGE-L, and BERT, alongside a novel metric, LLM-Accuracy, which uses a LLM to holistically evaluate the accuracy of the feedback.

The zero-shot evaluation results revealed that existing models performed poorly, primarily due to their turn-based nature and lack of domain-specific knowledge. Fine-tuning the models on QEVD improved performance but still lagged behind the proposed Stream-VLM, especially in terms of providing timely feedback.

Results and Implications

Stream-VLM outperformed other models, achieving higher scores in both fluency and temporal accuracy metrics. The model's ability to provide appropriate feedback at the right time, as illustrated in qualitative examples, underscores its potential for advancing real-world applications in fitness coaching.

The research has profound implications for the development of vision-LLMs:

1. Practical Applications: The ability to provide timely and contextually appropriate feedback can significantly enhance real-world applications like fitness coaching, rehabilitation, and remote training.
2. Model Design: Integrating specialized vision backbones with LLMs and employing action tokens can serve as a blueprint for developing interactive AI systems in other domains requiring situated interaction.
3. Future Research: The findings open avenues for exploring end-to-end training of domain-specific interactive vision models and incorporating additional modalities like speech input for a more holistic interaction.

Conclusion

Panchal et al.'s work represents a significant step towards developing real-time interactive AI systems. While the presented model and dataset address many challenges, the paper also highlights the limitations and potential biases in current models, emphasizing the need for further research in this area. The QEVD dataset and Stream-VLM model provide a robust foundation for future studies aiming to enhance the capabilities of vision-LLMs in real-world, situated interactions.