CoachMe: Multi-Modal AI Coaching Systems
- CoachMe is a family of AI coaching systems that leverage multimodal inputs and expert references to deliver corrective, context-aware feedback.
- It uses reference-based comparison and structured pedagogical workflows to translate observed behavior into actionable coaching guidance.
- Evaluation across domains demonstrates enhanced task alignment, reduced errors, and improved user engagement with both real-time and session-level feedback.
CoachMe appears in recent research both as a specific reference-based motion instruction model for sport technique analysis and as a broader architectural pattern for AI coaching assistants that convert multimodal observations into targeted guidance. Across these systems, camera video, pose trajectories, dialogue histories, diaries, route traces, or wearable audio streams are paired with reference exemplars, staged pedagogical workflows, or explicit evaluation rubrics so that the output is not merely description or scoring, but corrective, context-sensitive coaching (Yeh et al., 15 Sep 2025, Zuo et al., 10 Aug 2025, Chen et al., 22 Jun 2025).
1. Scope and domain coverage
CoachMe is not confined to a single application domain. The underlying idea recurs across embodied motor skill learning, academic writing, presentations, health counselling, walking support, speech habit intervention, tactical sports explanation, and negotiation coaching. In each case, the system mediates between observed behavior and an expert, pedagogical, or domain-specific standard, then returns actionable feedback.
| Domain | Representative system | Key coaching mechanism |
|---|---|---|
| Strength and mobility form | FormCoach (Zuo et al., 10 Aug 2025) | User–reference video comparison with concise corrective feedback |
| Sport motion instruction | CoachMe (Yeh et al., 15 Sep 2025) | Reference-based motion-to-instruction generation from temporal and physical differences |
| Academic writing | CoachGPT (Chen et al., 22 Jun 2025) | 11-stage scaffolding workflow with criteria-based feedback |
| Presentation practice | PresentCoach (Chen et al., 19 Nov 2025) | Ideal Presentation Agent, Coach Agent, and Audience Agent |
| Health counselling | COACH + QUORUM (Ng et al., 9 Mar 2026) | Diary-grounded counselling with retrieval over validated medical knowledge |
| Health dialogue coaching | DACT (Long et al., 7 May 2026) | Dual-agent co-training with MI-aware preference optimization |
| Walking companion | SmartWalkCoach (Zhang et al., 14 May 2026) | GeographyAgent, AccompanyAgent, and SummaryAgent |
| Speech habit intervention | WSCoach (Youpeng et al., 6 Jul 2025) | Near-real-time auditory feedback through smart glasses |
| Tactical and match coaching | PanoCoach (Kang et al., 2024), CoachAI (Hsu et al., 2019) | Mixed-reality tactical rehearsal and microscopic match analysis |
This breadth shows that CoachMe is best understood as a family of coaching systems rather than a single modality. Some variants operate synchronously in real time, such as camera-based exercise correction or wearable speech feedback, while others are session-level systems that summarize, evaluate, or scaffold longer tasks such as writing, presentations, or walking plans. A plausible implication is that the defining property of CoachMe is not the sensor or model family, but the explicit coupling of observation, interpretation, and corrective action.
2. Core architectural principles
A central architectural pattern is reference-based comparison. FormCoach compares a user’s movement against an expert demonstration and prompts VLMs to generate corrections in at most 15 words, using synchronized clips standardized to a mean window of approximately 3.8 seconds at 224×244 and 30 FPS (Zuo et al., 10 Aug 2025). The sport-specific CoachMe model formalizes this more explicitly: it aligns a learner video to a reference by selecting the start index
then computes concept differences and generates instruction from pooled motion and difference tokens (Yeh et al., 15 Sep 2025). In both cases, “correctness” is grounded in an external exemplar rather than inferred from generic captioning alone.
A second pattern is pedagogical decomposition. CoachGPT operationalizes educator instruction as 11 sequential stages ranging from pre-writing and resource identification to thesis, outline, paragraph construction, revising, word choice, grammar, and finalization (Chen et al., 22 Jun 2025). MetaCLASS performs a related decomposition for metacognitive tutoring, but at turn level rather than document stage level: it defines 11 coaching moves, including MONITOR_GOAL, CHECK_PROGRESS, STRATEGY_ALTERNATIVE, PROMPT_RESOURCE, and NO_INTERVENTION, with a target distribution in which NO_INTERVENTION should occupy 35–50% of coach turns (Liu et al., 2 Feb 2026). The shared principle is that coaching is treated as structured decision-making, not unconstrained generation.
A third pattern is closed-loop multi-agent interaction. PresentCoach instantiates an observation–practice–feedback cycle through an Ideal Presentation Agent that generates exemplar presentation videos, a Coach Agent that evaluates user recordings, and an Audience Agent that translates analytic signals into listener-centric reactions (Chen et al., 19 Nov 2025). DACT extends this logic to health coaching by co-training both coach and client simulator, optimizing the coach via Pareto-dominant preference pairs over Cultivating Change Talk, Softening Sustain Talk, and Empathy, while the client is trained adversarially through reversed preferences (Long et al., 7 May 2026). In a different direction, style-conditioned “coachable agents” define behavior as a fixed task plus a runtime-modifiable style term,
so an end user can steer how a task is executed without retraining the policy (Capobianco et al., 1 Jul 2026).
Taken together, these architectures treat coaching as control over trajectories: physical trajectories, dialogue trajectories, learning trajectories, or style trajectories. This suggests that CoachMe research increasingly replaces one-shot recommendation with policy-like sequencing.
3. Multimodal sensing, data representations, and interaction loops
The embodied variants of CoachMe rely on dense multimodal observation. FormCoach captures live user video side-by-side with an expert reference, synchronizes the two streams, and feeds paired clips plus user goals into prompted VLMs. The user can specify preferences such as “focus on my knee alignment,” and the system can render short corrections on screen or via text-to-speech (Zuo et al., 10 Aug 2025). Its evaluation dataset consists of 1,700 expert-annotated user–reference video pairs spanning 22 strength and mobility exercises, sampled from Fit3D-derived clips.
The reference-based sport CoachMe model uses a more elaborate motion stack. Its Concept Difference module adopts a ResNet-50-based encoder from CARL, while the Human Pose Perception module uses HybrIK to estimate 22 3D joints and then applies three GCN-based submodules—Pose Understanding, Pose Extraction, and Pose Attention—to construct local-global motion tokens (Yeh et al., 15 Sep 2025). Instruction generation is performed by a T5-base LLM with 223M parameters, with LoRA adaptation for sport-specific finetuning. The training corpora include HumanML3D with 23,384 train and 4,384 test motions, a figure skating dataset with 177 train and 40 test videos plus 449/64 ground-truth clips, and a boxing dataset with 163 train and 41 test videos.
Other CoachMe variants demonstrate that coaching datasets increasingly align behavior, language, and internal state. SimCoachCorpus records 29 humans driving in a simulator for approximately ninety minutes, with 15 coached participants and 14 self-practice participants, and includes over 20,000 concurrent feedback utterances, over 400 terminal feedback utterances, and over 40 hours of vehicle driving data synchronized with trajectories, map context, and survey measures (Sumner et al., 18 Sep 2025). WSCoach uses Huawei audio smart glasses, Faster-Whisper, and pyttsx4 to detect user-specified unwanted words and return compressed spearcon feedback with average detection latency 0.81 s, average playback duration 0.28 s, and end-to-end delivery of approximately 1.1 s (Youpeng et al., 6 Jul 2025).
A broader systems point follows from these examples. CoachMe implementations do not only observe task outputs; they instrument temporal context, alignment structure, and often repeated measures of user state. That instrumentation is what makes coaching more than after-the-fact scoring.
4. Evaluation paradigms and empirical performance
Evaluation in CoachMe research is notably heterogeneous, because coaching quality is partly technical and partly experiential. FormCoach evaluates model-generated corrections with a deterministic GPT-4.1 rubric over Accuracy, Actionability, and Hallucination. The best reported model, GPT-4.1, achieved 58.2% Accuracy, 94.4% Actionability, and 74.26% Hallucination, while even the strongest systems remained substantially below human-level coaching robustness (Zuo et al., 10 Aug 2025). Exercise-wise analysis further showed higher mean accuracy on simpler upper-body motions such as Barbell Shrug and Hands Up Rotate, and lower mean accuracy on compound, occlusion-heavy movements such as Burpees, Barbell Pull Up, and Mule Kick.
The sport-specific CoachMe model uses both automatic and expert evaluation. On G-Eval consistency, it improved over GPT-4o from 1.39 to 1.83 on figure skating, a relative improvement of 31.6%, and from 1.39 to 2.20 on boxing, a relative improvement of 58.3% (Yeh et al., 15 Sep 2025). In human evaluation, CoachMe obtained the highest proportion of “Good” ratings in both domains: 26.6% in figure skating and 56.0% in boxing. The same study also reported that predicted error segments improved instruction quality relative to using the whole aligned video, and that skeleton-based difference tokens outperformed RGB-based alternatives.
Human-subject studies in adjacent CoachMe systems show that purely technical metrics are insufficient. PresentCoach evaluated 24 non-native English speakers and reported significant PRCS pre–post improvement for the PresentCoach+PPT condition with , while PPT-only did not show a significant change; workload was moderate at , on a 7-point NASA-TLX scale, and usability was high at , on a 5-point SUS subset (Chen et al., 19 Nov 2025). SmartWalkCoach used a two-period AB/BA crossover study with and found that Information+Motivation improved positive feelings and user experience relative to Information-only, with Cohen’s 0 and 1 respectively (Zhang et al., 14 May 2026). DACT, by contrast, evaluates utterance-level MI quality: its 4-condition average mean3 reached 2, while average anti-pattern rate fell to 3, outperforming GPT-based baselines on both coaching quality and safety-oriented linguistic behavior (Long et al., 7 May 2026).
These evaluation schemes reveal a consistent methodological feature: CoachMe systems are increasingly judged simultaneously on task alignment, actionability, safety, and user-perceived usefulness. No single metric covers all four.
5. Reliability, safety, and human factors
A recurrent finding is that stylistic fluency is not equivalent to coaching competence. In the multi-turn coaching study “Substance over Style,” users preferred facilitative agents, with 67.7% ranking a facilitative system first overall, and the Expert-Facilitative agent achieving a 61.29% win rate over rank-1 plus rank-2 placements (Srinivas et al., 25 Mar 2025). The same study found that users highly value core functionality and regard stylistic components without core components negatively. Interrogative agents also had markedly higher forced-ending rates, reaching 58.1% for Base-Interrogative and 71.0% for Expert-Interrogative.
A related risk is compulsive intervention. MetaCLASS reports that effective metacognitive tutoring requires NO_INTERVENTION in 41.7% of cases, yet large models predicted NO_INTERVENTION only 4.2% of the time and severely over-predicted high-intervention moves (Liu et al., 2 Feb 2026). This matters because over-assistance can suppress self-regulated learning, just as over-protective shared control can induce over-reliance in motor-skill coaching.
Hallucination remains a major reliability problem in embodied and health coaching. FormCoach’s high hallucination rates are explicitly identified as a safety concern, especially because users may act on nonexistent or misidentified form issues (Zuo et al., 10 Aug 2025). In diary-grounded health counselling, QUORUM showed convergence and divergence across users, experts, and developers: users rated alignment and likelihood to act at 3.90 and tone at 4.32, experts rated contextual correctness at 4.02 and tone at 3.73, while developer audits reported faithfulness 0.79, completeness 0.97, and hallucination 0.22 (Ng et al., 9 Mar 2026). The same study showed higher agreement between experts and users than between either group and an LM auto-rater on several subjective dimensions.
Privacy and consent are not secondary concerns in this literature. PresentCoach explicitly recommends consent gating, provenance tracking, opt-out and delete controls, liveness checks, watermarking cloned audio, rate limiting, and encryption for voice cloning workflows (Chen et al., 19 Nov 2025). WSCoach, which performs always-on speech monitoring, emphasizes immediate deletion of raw audio after transcription, minimal metadata retention, and future integration of diarization and speaker verification to avoid bystander-triggered feedback (Youpeng et al., 6 Jul 2025). In health coaching, DACT recommends clinician oversight, escalation workflows on suicidality or acute distress, synthetic personas for co-evolution training, and anonymization for any real-user preference data (Long et al., 7 May 2026).
The broader implication is that CoachMe systems are now evaluated not only by whether they can coach, but by whether they know when to abstain, when to disclose uncertainty, and how to protect the user while doing so.
6. Emerging directions
Several future directions recur across the literature. One is multimodal fusion beyond video-only reasoning. FormCoach explicitly identifies pose and IMU integration as a path to viewpoint-independent kinematics and lower hallucination rates (Zuo et al., 10 Aug 2025). The sport-specific CoachMe paper similarly points toward explicit physical constraints, wearable or IMU streams, broader sport coverage, lower-latency deployment, and multilingual guidance (Yeh et al., 15 Sep 2025).
A second direction is richer feedback embodiment. PresentCoach proposes overlays, spatial audio, body-language avatars, progress-aware feedback, and spaced repetition with micro-drills (Chen et al., 19 Nov 2025). PanoCoach already shows how synchronized 2D boards, 3D views, and first-person VR can anchor tactical explanation in embodied perception during soccer training, while CoachAI extends that logic to badminton through microscopic competition data, AR/VR visualization, and connected serving machines (Kang et al., 2024, Hsu et al., 2019).
A third direction is personalized timing and adaptive orchestration. SmartWalkCoach argues for sparse, context-aware interventions keyed to pace drops, milestones, intersections, and learned receptivity, and suggests contextual bandits or reinforcement learning for long-term optimization of prompt timing and content (Zhang et al., 14 May 2026). The RL-based motor coaching framework formalizes this in stronger terms: the coach’s objective is not immediate assisted performance, but the learner’s future independent competence, optimized through selective scaffolding and productive failures (Wang et al., 24 Jun 2026).
A fourth direction is agentic retrieval and semantic querying over coaching records. The athlete profiling framework “Digitizing Coaching Intelligence” combines MediaPipe, Llama-4-scout, an LLM-as-a-Judge correction loop, and dual persistence into relational storage plus ChromaDB, while its 3×3 Smart Grid temporal chunking reduces multimodal processing overhead by approximately 88.9% (Ghosal et al., 26 Jun 2026). This suggests an infrastructure shift in which CoachMe is not only a feedback generator but also a searchable knowledge layer over prior sessions, athletes, and qualitative observations.
Taken together, these trajectories indicate a move from isolated assistant functions toward persistent coaching ecosystems: multimodal, retrieval-backed, self-auditing, and increasingly explicit about pedagogy, safety, and user control.