VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition
Abstract: Videos are unique in their ability to capture actions which transcend multiple frames. Accordingly, for many years action recognition was the quintessential task for video understanding. Unfortunately, due to a lack of sufficiently diverse and challenging data, modern vision-LLMs (VLMs) are no longer evaluated on their action recognition capabilities. To revitalize action recognition in the era of VLMs, we advocate for a returned focus on domain-specific actions. To this end, we introduce VideoNet, a domain-specific action recognition benchmark covering 1,000 distinct actions from 37 domains. We begin with a multiple-choice evaluation setting, where the difference between closed and open models is stark: Gemini 3.1 Pro attains 69.9% accuracy while Qwen3-VL-8B gets a mere 45.0%. To understand why VLMs struggle on VideoNet, we relax the questions into a binary setting, where random chance is 50%. Still, Qwen achieves only 59.2% accuracy. Further relaxing the evaluation setup, we provide $k\in{1,2,3}$ in-context examples of the action. Some models excel in the few-shot setting, while others falter; Qwen improves $+7.0\%$, while Gemini declines $-4.8\%$. Notably, these gains fall short of the $+13.6\%$ improvement in non-expert humans when given few-shot examples. Finding that VLMs struggle to fully exploit in-context examples, we shift from test-time improvements to the training side. We collect the first large-scale training dataset for domain-specific actions, totaling nearly 500k video question-answer pairs. Fine-tuning a Molmo2-4B model on our data, we surpass all open-weight 8B models on the VideoNet benchmark.
Summary
- The paper introduces VideoNet, a benchmark featuring 1,000 actions in 37 domains to test fine-grained domain-specific recognition.
- It employs a multi-stage human annotation pipeline and LLM-assisted filtering, achieving a 97.6% verification accuracy for high-quality labels.
- Evaluations reveal that VLMs struggle with hard-negative distractors, emphasizing the need for targeted training data and improved motion understanding.
Summary of "VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition"
Motivation and Background
Action recognition is central to video understanding, bridging perception and compositional reasoning in vision-LLMs (VLMs). Prior benchmarks focus heavily on either coarse-grained actions (e.g., "rock climbing"), limited sports domains, or fine-grained temporal attributes, providing little insight into the real-world applicability of action recognition in diverse domains. As a result, modern VLMs are rarely evaluated on their ability to distinguish domain-specific actions, inhibiting progress in advancing visual reasoning and multi-domain generalization.
The "VideoNet" benchmark addresses this gap by providing a systematic evaluation framework for domain-specific action recognition across 1,000 actions in 37 domains―spanning sports, crafts, dance, food, medical, beauty, and hobbies. The benchmark is constructed to rigorously test both perception of fine-grained motion and domain-level compositionality in realistic, cross-domain scenarios.
Figure 1: Video samples from all 7 categories and 37 domains in VideoNet, exemplifying benchmark breadth and visual diversity.
Benchmark Design and Data Collection
VideoNet's taxonomy leverages a top-down, category-driven approach, ensuring coverage of actions relevant to daily life, expert skills, and rapid-motion recognition. Actions and their definitions are sourced and augmented via LLM-assisted web search, ensuring high-fidelity, visually-grounded cues and minimizing domain-expert bias.
The multi-stage human annotation pipeline employs non-expert annotators for video collection, outlier detection, and trimming―each stage informed by HCI practices and majority voting for quality control. The resulting dataset consists of five well-trimmed clips per action, substantiated by 97.6% verification accuracy via domain experts, establishing robust ground truth labels.
Figure 2: Data collection pipeline: web search, clip outlier removal, and temporal trimming yield high-quality annotated clips.
Challenging hard-negative labels are constructed using LLMs and refined by reasoning models, avoiding confounding co-occurences and ensuring visually similar, compositional distractors. This design circumvents reliance on static cues, compelling models to attend to motion and nuanced action boundaries.
Kernel density estimation reveals a heavy-tailed distribution of clip durations, with most actions in the 5–12 second range, but long tails for domains like suturing and crochet.
Figure 3: VideoNet clip duration distribution, highlighting wide temporal diversity and contextual richness.
Evaluation Protocols
VideoNet provides two principal evaluation settings:
- Multiple-choice: Given a video and four plausible action labels (one positive, three hard negatives), models select the correct action—a protocol emphasizing domain-specific recognition.
- Binary few-shot: Models receive k in-context exemplars (k ∈ {1,2,3}), then evaluate whether subsequent videos contain the target action—testing in-context visual learning.
Figure 4: Q&A examples for both evaluation settings (multiple-choice and binary few-shot), capturing video-to-action mapping complexity.
A balanced test set (50% positives, 50% hard negatives) and random baseline (chance performance) are included for reference.
Baseline Model Performance and Ablations
Systematic evaluation of proprietary and open-weight VLMs revealed substantial gaps in domain-specific action recognition. In the multiple-choice setup, closed models (e.g., Gemini 3.1 Pro) attain 69.9% accuracy, while top open-weight models (Qwen3-VL-8B, Molmo2-8B) reach only ~45%. Binary 0-shot accuracies similarly expose deficiencies; open models lag behind human annotators, even under relaxed settings.
Ablation studies demonstrate that full-video inputs yield minimal gains in open VLMs versus single frames, indicating insufficient utilization of motion cues and an over-reliance on static bias. Adding action definitions provides negligible improvement, underscoring the challenge stems from visual mapping rather than lack of semantic prior.
Higher frame rates (fps) afford diminishing returns, with proprietary models plateauing. Test-time scaling via denser sampling does not adequately address motion understanding limitations.
Figure 5: Model accuracy as a function of video input type and frame sampling, showing weak temporal scaling.
Few-Shot In-Context Learning Analysis
Contrary to expectations set by LLMs in text, VLMs exhibit limited capacity to leverage few-shot visual demonstrations. Gains are modest (average +2.9 percentage points), with inconsistent model behavior: Qwen3-VL improves (+7%), Gemini 3.1 Pro declines (–4.8%). The lack of systematic benefit suggests in-context learning in VLMs remains immature, especially for video-based compositional reasoning.
Non-expert human annotators, conversely, benefit substantially from few-shot examples (+13.6%), far surpassing model few-shot improvement. This illustrates a significant gap between human visual learning mechanisms (potentially involving mirror neurons and multi-modal abstraction) and current VLM architectures.
Figure 6: Binary few-shot accuracy for VLMs and humans; humans rapidly generalize from few-shot visual input, models fail to exploit context fully.
Changing from hard negatives to random negatives further evidences the benchmark’s difficulty; both models and humans perform significantly better on random distractors, confirming that VideoNet’s negative options are precisely tuned to challenge fine-grained visual discriminability.
Figure 7: Positive vs. negative accuracy shifts with in-context examples—models display substantial bias and sensitivity.
Training Set Construction and Fine-Tuning Outcomes
To address the training-deficiency hypothesis, VideoNet introduces a large-scale, auto-labeled training set (~500k video Q&A pairs). Clips are localized via Gemini, then filtered using transcript-title alignment strategies and labeled at scale. Three principal filtering strategies balance quality and coverage: transcript match, strict title+transcript match, and title-oneclip match.
Fine-tuning Molmo2-4B on the most strict title-oneclip filter data yields an 11.5 point improvement in multiple-choice accuracy (53.5%), exceeding all open 8B models. Binary 0-shot accuracy similarly improves, reinforcing that domain-specific data is instrumental in closing the action recognition gap in VLMs.
Implications and Future Directions
VideoNet establishes a critical new benchmark for domain-specific action recognition, highlighting persistent deficits in VLM compositionality and motion understanding. The strong empirical results underscore the importance of domain-specific, high-quality training data and reveal meaningful avenues for model architecture innovation—particularly for grounding input motion in fine-grained action labels and improving in-context video learning.
Practically, VideoNet enables systematic evaluation of real-world action recognition, with implications for medical diagnostics, skill acquisition, sports analytics, and human-robot interaction. Theoretical implications include new challenges for multi-domain compositional generalization, temporal modeling, and efficient few-shot learning in visual LLMs.
As multimodal foundation models progress, VideoNet will serve as a standard for diagnosing visual reasoning capabilities, guiding future research toward bridging the perceptual and semantic gaps inherent in current video understanding architectures.
Conclusion
VideoNet provides a rigorous benchmark and accompanying large-scale dataset for domain-specific action recognition, exposing substantial shortcomings in current vision-LLMs and offering a scalable pathway for improvement via targeted training data. The results demonstrate that high-quality, domain-specific annotation—combined with carefully designed negative distractors and comprehensive evaluation settings—can catalyze advances in VLM compositionality, motion understanding, and visual in-context learning.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
VideoNet: A simple explanation
What is this paper about?
This paper introduces VideoNet, a big new test and training resource for teaching computers to recognize “domain‑specific” actions in videos. Domain‑specific actions are moves that mean something in a particular field—like a “lutz jump” in figure skating, a “suturing knot” in medicine, or a “rear delt fly” in the gym. These are harder than general actions (like “running” or “clapping”) because you have to notice tiny details and the exact order of movements.
The authors show that today’s video‑language AI systems are not very good at these kinds of actions. They also build tools and data that help models improve.
1) Brief overview
The paper’s main goal is to bring back serious testing of action recognition for modern AI by focusing on domain‑specific actions. The authors:
- Create a benchmark called VideoNet with 1,000 different actions across 37 domains (like sports, dance, crafts, medical, food, beauty, and hobbies).
- Test popular AI models on this benchmark in two ways: a multiple‑choice quiz and a yes/no quiz (binary).
- Study whether giving a few examples at test time helps models learn (few‑shot learning).
- Build a large training dataset (hundreds of thousands of clips) and fine‑tune a smaller open model that ends up beating larger open models.
2) Key objectives in simple terms
The paper asks:
- Can today’s AI models recognize precise, domain‑specific actions in videos?
- Do they get better if we give them a few example videos during the test?
- Are the models actually using motion in the video, or just guessing from single frames and background clues?
- If we train models on a lot of domain‑specific action clips, how much can they improve?
3) How did they do it? (Methods explained simply)
Building the benchmark
- Action list: They chose 1,000 actions across 37 domains. They started with trusted expert sources (like sport or craft communities) and used AI plus web search to write clear, visual definitions for each action.
- Video collection: Crowd workers found video clips that show each action. Multiple people checked each clip, and another person fixed the start and end times so the clip shows just the action. One of the authors did a final pass to make sure everything was right.
- Expert check: In a subset of domains, experts checked 620 clips and found the labels were correct about 97% of the time.
- Hard negatives: For each action, they also picked “look‑alike” actions that are easy to confuse (for example, two similar basketball dunks). These hard choices make the test fair and challenging, so models can’t just guess from the scene or objects.
Two ways to test models
- Multiple‑choice: Each question shows a video and four similar action names from the same domain. The model must pick the right one.
- Binary few‑shot: Each question is yes/no (“Is this action X?”). The model also gets 0, 1, 2, or 3 short example videos of action X to learn from before answering.
Building a large training set (automatic)
To help models learn, the authors also created a much bigger training set:
- They crawled millions of videos from the web using action names (like “how to laser flip”).
- A model (Gemini) was used to find the time ranges where an action occurs in each video (localizing “when” without needing to name the action exactly).
- They labeled clips by matching the action name to the video’s title and transcript (the words spoken in the video), using three strategies from strict to broader matches.
- This pipeline produced about 160,000 to 500,000 labeled clips, which were turned into nearly 500,000 video question‑answer pairs.
Fine‑tuning a model
- They fine‑tuned an open, 4‑billion‑parameter model (Molmo2‑4B) on this new training data.
- During training, they sample a few frames per second from each clip and feed timestamps as text so the model knows the temporal order.
4) Main findings and why they matter
Here are the most important results, summarized for clarity:
- Performance on the benchmark:
- Multiple‑choice (4 options; random guess ≈ 25%): The best big “closed” models (company‑hosted) scored around 70%. The best open‑source models around 45%.
- Binary (yes/no; random guess = 50%): Open models scored around 59%; closed models around 72%; non‑expert humans around 69%.
- Few‑shot learning (giving 1–3 example videos at test time):
- Humans improved a lot with three examples (+13.6 percentage points, reaching 82.7% with action definitions).
- Models improved only a little on average (+~3 points), and some even got worse. This means most current video models don’t learn very well from a few visual examples.
- Are models using motion?
- Many open models barely improved when given the whole video versus a single frame, suggesting they rely too much on static cues (like the court or equipment) instead of the actual movement.
- Providing written action definitions didn’t help much either. The issue isn’t “what is the action?” knowledge; it’s connecting that knowledge to the motion in the video.
- Sampling more frames per second gave only small gains, so just “showing more frames” doesn’t solve the problem.
- Training helps more than test‑time examples:
- After fine‑tuning the 4B model on the new domain‑specific data, it jumped by +11.5 points in multiple‑choice (to 53.5%) and to 66.6% on binary zero‑shot—beating all open 8B models. This shows that targeted training data can matter more than showing examples at test time.
- Hard vs. random negatives:
- When the alternatives were easy (random), both humans and models did much better. With the hard look‑alikes, accuracy dropped—especially for humans—confirming that VideoNet really tests subtle, real‑world differences.
Why this matters: It shows modern vision‑LLMs struggle with fine, domain‑specific motions that experts care about—exactly the skills needed for real tasks like coaching, training, or medical guidance.
5) What does this mean for the future? (Implications)
- Better training beats quick fixes: Giving a few video examples at test time isn’t enough today. Models learn more reliably when you train them on lots of well‑labeled, domain‑specific clips.
- Focus on motion and reasoning: Models need to pay attention to the exact order and details of movements, not just static objects or backgrounds.
- Real‑world impact: With improvements, these systems could help beginners learn sports techniques, coaches review athlete form, hobbyists master tricks, or students study medical procedures safely.
- A foundation for research: VideoNet gives researchers a challenging, diverse benchmark and a recipe for collecting more domain‑specific training data—fueling progress in video understanding that is useful outside the lab.
In short, this paper shows that recognizing expert‑level actions in video is still a hard problem for AI, but targeted data and training can close the gap—and opens the door to practical, helpful video understanding tools.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper advances domain-specific action recognition with a new benchmark and training data pipeline, but several aspects remain underexplored or uncertain. The following concrete gaps can guide future research:
- Dataset coverage and representativeness
- Lack of quantified geographic, cultural, and production-style diversity; web-sourced videos may skew toward anglophone, western, and tutorial-style content.
- Uneven domain/label coverage (long-tail actions) is acknowledged but not measured; only 5 clips per action may limit statistical robustness and within-class visual variability.
- “Food” category appears object/context-driven; a curated “hard” subset is suggested but not provided, leaving unclear how to isolate action-centric difficulty across domains.
- Label quality, definitions, and negatives
- Expert verification covers only 7 of 37 domains and 620/5,000 clips; applicability of 97% accuracy to all domains is uncertain; no inter-annotator agreement statistics are reported.
- Definitions are LLM-generated and web-checked but not fully audited for all actions; the impact of definitional inaccuracies on model/human performance is unknown.
- Hard negative generation uses LLMs and a “reasoning model,” but cross-action co-occurrence and ambiguity risks persist; validity and difficulty calibration across domains are not systematically measured.
- Manual final pass by a single author may introduce subjective bias; no independent second-pass audit reported.
- Potential train–test contamination and reproducibility
- Both benchmark and training data are web-sourced; no explicit near-duplicate or source-level deduplication between training crawls and test clips is described.
- The use of proprietary Gemini for localization in the training pipeline raises reproducibility concerns and potential leakage if Gemini (or other closed models) was trained on similar web content.
- Exact release scope of the large-scale training data (clips, timestamps, transcripts) is unclear; licensing/permissions for redistribution are not discussed.
- Evaluation design and fairness
- Cross-model comparisons use model-specific fps and frame limits (“recommended strategies”), confounding fairness; sensitivity to sampling choices is minimally explored (main fps ablation shown only for GPT-5.4).
- Accuracy is the sole metric; no calibration, confidence, or error-type analyses (e.g., systematic confusions between specific action pairs) are reported.
- Multiple-choice with textual options may cue answers via lexical/semantic priors; the extent to which models exploit text priors vs. video evidence is not quantified.
- Binary few-shot setup fixes in-context examples (first three clips) without selection strategy analysis; ordering, similarity, and diversity effects are untested.
- Few-shot and in-context learning
- Only k≤3 video demonstrations are examined; no exploration of mixed video+text demonstrations, captions, or retrieval-augmented exemplars that might enhance video ICL.
- Large variance across models (some degrade with more examples) is not analyzed mechanistically; factors like context window limits, frame budgets, and attention allocation remain open.
- No study of exemplar selection policies (e.g., diversity, hardest positives, prototypical clips) or their impact on in-context generalization.
- Motion understanding vs. static bias
- Single-frame vs. video comparisons suggest static biases in open models, but deeper analyses (e.g., optical flow, trajectory tokens, long-range temporal modeling, causal motion cues) are limited.
- FPS scaling shows diminishing returns in one model; broader, model-agnostic studies on spatiotemporal tokenization, clip length, and temporal resolution trade-offs are missing.
- Modeling and ablations
- Little exploration of architectural choices that could improve action sequencing and compositionality (e.g., temporal order verification, event-level memory, hierarchical modeling).
- Timestamp text encoding is used during fine-tuning but not ablated; its contribution versus alternative temporal encodings remains unknown.
- The efficacy of audio or audio-visual fusion is not explored, despite many domain-specific actions potentially having distinctive sound cues.
- Generalization and transfer
- Generalization to unseen domains/actions is not evaluated (no leave-one-domain-out or novel-action tests).
- Robustness to challenging conditions (egocentric footage, occlusion, crowd scenes, low light, camera motion, multiple simultaneous actors) is not systematically assessed.
- Cross-dataset transfer to other fine-grained action benchmarks (e.g., Ego-Exo4D subsets, ActionAtlas, domain-specific datasets) is not reported.
- Action complexity and temporal structure
- Benchmark focuses on trimmed clips; models are not evaluated on temporal localization of actions in untrimmed videos or on multi-action detection.
- Many domain-specific actions entail ordered sub-movements; explicit evaluation of temporal compositionality (order/phase sensitivity, pre/post-conditions) is absent.
- Human evaluation scope
- Non-expert human evaluations cover 698 questions; lack of expert baselines across all domains, inter-rater reliability metrics, and analysis of where non-experts systematically fail.
- Effects of learning curves, time-on-task, and instruction/definition quality on human performance are not studied.
- Data scaling and selection
- Smaller but “stricter” training dataset outperforms larger ones; the quality–quantity trade-off is not characterized across more scales and domains.
- No investigation of curriculum learning, active data selection, or uncertainty-driven filtering to balance quality and coverage, particularly for long-tail domains.
- Ethics, legal, and societal considerations
- Legal status of scraped videos (rights, licenses, consent), especially for medical or identifiable subjects, is not detailed.
- Potential demographic and cultural biases in video sources and action taxonomies are not analyzed; risk of reinforcing stereotypes remains unaddressed.
- Explainability and safety
- While some prompts encourage “reasoning,” there is no evaluation of rationale faithfulness or whether models genuinely use motion evidence versus spurious cues.
- No assessment of failure modes that could cause safety issues (e.g., misrecognition in medical or gym contexts) or guidelines for safe deployment.
- Practical deployment questions
- Real-time performance, latency vs. accuracy trade-offs, and compute costs for varying fps/frame budgets are not analyzed.
- Impact of on-screen text, watermarks, or subtitles (OCR cues) on model decisions is untested, leaving open the risk of shortcut exploitation.
These gaps suggest concrete directions: rigorous deduplication and bias audits; standardized evaluation protocols and deeper temporal/multimodal ablations; principled few-shot exemplar selection; out-of-domain generalization tests; and ethical/legal clarity and release details for training corpora.
Practical Applications
Overview
The paper introduces VideoNet, a domain-specific action recognition benchmark (1,000 actions, 37 domains) and a scalable data pipeline that (i) localizes actions in web videos, (ii) labels clips via titles/transcripts, and (iii) generates hard negatives using LLMs. It shows that fine-tuning a compact open VLM (Molmo2-4B) on ~160k–500k auto-labeled clips surpasses open 8B models, and that current VLMs underutilize few-shot video demonstrations compared to humans.
Below are practical applications derived from these findings, organized by deployment horizon. Each item summarizes potential tools/products/workflows and key assumptions/dependencies.
Immediate Applications
These can be piloted now with existing models, the VideoNet benchmark, and the paper’s data/labeling pipelines.
- Action-centric model evaluation and procurement (Software/AI platforms)
- What: Use VideoNet to assess and compare VLMs’ domain-specific action recognition before integrating into products.
- Tools/workflows: Benchmark harness, per-domain scorecards, hard-negative stress tests for spurious cue detection.
- Dependencies/assumptions: Acceptance of benchmark representativeness; model evaluation budgets; license compliance for benchmark use.
- Low-cost video dataset bootstrapping via “localize-then-label” (Software, Media, Robotics)
- What: Replicate the paper’s training data pipeline to build internal, domain-specific action datasets without domain experts.
- Tools/workflows: Video crawling → VLM-based localizer (e.g., Gemini-like) → WhisperX transcripts → title/transcript label filters (Transcript, Strict, Title-One-Clip).
- Dependencies/assumptions: Availability of videos with usable transcripts/titles; rights to crawl/use content; reliable localizer; compute resources.
- Action-aware video indexing and search (Media/Streaming, Sports, Ed-tech)
- What: Tag catalogs by specific actions (e.g., “double lutz,” “giant swing,” “lat pulldown”) to power search, recommendations, and highlight reels.
- Tools/workflows: Fine-tuned 4B model as an action tagger; action-index API; hard-negative-tuned validation to reduce scene-bias.
- Dependencies/assumptions: Tolerable precision/recall for production; scalable inference; content rights.
- Progress tracking features in learning apps (Sports/Fitness, Dance, Crafts, Cooking)
- What: Detect whether a learner performed the target action to unlock next lessons or log practice milestones (binary detection, not quality grading).
- Tools/workflows: Mobile/server inference using fine-tuned compact VLMs; lesson flows keyed to detected actions; user consent UI.
- Dependencies/assumptions: Controlled camera viewpoints; acceptable false positives/negatives; privacy safeguards.
- Training checklists and compliance in low-stakes settings (Hospitality, Makerspaces, Lab classes)
- What: Verify that required procedural steps occurred (e.g., espresso tamping, pour-over steps) during training sessions.
- Tools/workflows: Action checklists with timestamps; instructor dashboards; audit trail export.
- Dependencies/assumptions: Stable camera setups; domain similarity to dataset; human-in-the-loop review.
- Trust & Safety triage for high-risk actions (Online Platforms)
- What: Flag videos with risky domain-specific actions for human review (e.g., dangerous stunts, unsafe gym techniques).
- Tools/workflows: Action-based filters tuned on hard negatives; escalation queues for moderators.
- Dependencies/assumptions: High-recall thresholds; clear policy definitions; calibration to minimize overblocking; privacy impact assessment.
- Medical education content organization (Healthcare Education)
- What: Index neuro-exam and procedural training recordings by maneuvers for quick retrieval (e.g., “Romberg,” “Babinski sign”).
- Tools/workflows: Domain adaptation + faculty-reviewed validation sets; curriculum-aligned tags.
- Dependencies/assumptions: Non-diagnostic use; IRB/privacy compliance; generalization to local curricula.
- Robotics dataset curation from web videos (Robotics)
- What: Pre-filter demonstrations by action type (e.g., “unscrewing,” “folding”) to accelerate downstream imitation or segmentation studies.
- Tools/workflows: Localize-then-label pipeline; action filters to cluster demonstrations; manual refinement.
- Dependencies/assumptions: Domain shift between human web videos and robot embodiments; additional state estimation required.
- Research on video in-context learning and temporal reasoning (Academia)
- What: Use VideoNet’s binary few-shot setting and ablations to study why VLMs underuse video demonstrations and motion cues.
- Tools/workflows: Released dataset/splits; ablation scripts; Molmo2-4B fine-tuning recipe as a strong small-model baseline.
- Dependencies/assumptions: Compute access; compliance with dataset licenses; reproducibility practices.
- Hard-negative generation for stress testing (AI Evaluation)
- What: Apply the LLM-driven hard-negative generation with co-occurrence filtering to other domains/datasets for robust evaluation suites.
- Tools/workflows: Prompt templates + reasoning filter; ambiguity screening; per-domain confusion sets.
- Dependencies/assumptions: High-quality LLMs/reasoners; expert spot checks; maintenance for evolving taxonomies.
Long-Term Applications
These require further research, scaling, validation, or regulatory pathways before deployment.
- Real-time technique feedback and action quality assessment (Sports, Fitness, Performing Arts)
- What: Beyond detecting “which action,” provide graded feedback on correctness/order/nuance (e.g., squat depth/valgus, figure skating jump edges).
- Tools/workflows: Couple action recognizers with pose tracking and temporal scoring; AR overlays; personalized feedback loops.
- Dependencies/assumptions: Robust multi-view capture; low-latency on-device inference; liability management; validation by coaches/experts.
- Automated surgical/clinical skill assessment and credentialing (Healthcare)
- What: Evaluate adherence to procedural steps and technical skill from video of maneuvers (e.g., suturing, neuro exams) to support training and QA.
- Tools/workflows: Protocol-aware action graphs; error taxonomy; integration with skills labs and EMR-lite training systems.
- Dependencies/assumptions: Prospective clinical trials; FDA/CE approvals; bias and safety audits; strict privacy and access controls.
- Teach-at-runtime video ICL: user-defined actions without retraining (Enterprise Video Analytics, Robotics)
- What: Let users define new actions by providing a few example clips at inference time for immediate deployment.
- Tools/workflows: Architectures purpose-built for video ICL with longer temporal context and memory; retrieval-augmented adapters.
- Dependencies/assumptions: Advances in temporal representation learning; longer context windows; robust generalization to negatives.
- Robotics imitation learning from web videos at scale (Robotics, Logistics, Manufacturing)
- What: Extract fine-grained action segments and ordering constraints as supervision for policies (e.g., assembly step sequences).
- Tools/workflows: Action segmentation + correspondence to robot affordances; 3D scene understanding; policy learning with temporal constraints.
- Dependencies/assumptions: Action-to-actuator mapping; domain adaptation; safety guarantees during deployment.
- Automated compliance auditing from CCTV/training video (Food Safety, Pharma, Manufacturing, Insurance)
- What: Verify protocol sequences (e.g., sanitation steps, PPE donning) and generate exception reports for QA and claims.
- Tools/workflows: Action-sequence validators; privacy-preserving analytics; auditor dashboards with explainability.
- Dependencies/assumptions: Worker consent and privacy laws; high-accuracy requirements; stakeholder buy-in and change management.
- Advanced sports analytics and broadcasting (Sports/Media)
- What: Real-time detection of named skills/moves for live stats, coaching breakdowns, and personalized highlight generation.
- Tools/workflows: Low-latency inference at broadcast edge; action-aware clipping; integration with coaching tools.
- Dependencies/assumptions: Rights and MAM integration; robustness to broadcast edits/graphics; latency budgets.
- Adaptive multimodal tutoring for skills (Education)
- What: Personalized pacing and remediation based on recognized actions (e.g., crochet stitches, painting techniques, barista steps).
- Tools/workflows: Curriculum graphs linked to action detectors; multimodal hints; mastery-based progression.
- Dependencies/assumptions: Diverse, inclusive datasets; device variability; accessibility considerations.
- Domain-specialized compact VLMs for on-device use (Edge AI, Mobile)
- What: Train 4B–8B models per vertical (e.g., sports coaching, crafts) using the paper’s pipeline for private, fast, on-device inference.
- Tools/workflows: Domain-specific pretraining; quantization and distillation; mobile runtimes.
- Dependencies/assumptions: Sufficient high-quality data per vertical; licensing for training data; performance tuning.
- Policy and certification frameworks for video AI (Public Policy, Standards)
- What: Adopt domain-specific benchmarks with hard negatives as part of certification for high-stakes video AI (e.g., medical, safety).
- Tools/workflows: Standardized evaluation protocols; transparency reporting on motion vs. static-cue reliance; bias audits.
- Dependencies/assumptions: Multi-stakeholder consensus; open, rights-cleared test suites; governance for updates.
- Privacy-preserving action recognition (Consumer Devices, Healthcare)
- What: Federated/on-device training and inference for action recognition in homes, gyms, or clinics to minimize video sharing.
- Tools/workflows: Federated learning; differential privacy; on-device compact models; secure enclaves.
- Dependencies/assumptions: Efficient hardware; reliable personalization without leakage; regulatory compliance.
Notes on feasibility across applications:
- Performance varies by domain; “Food” actions may be object-detection dominated, while “Sports/Medical” require fine-grained temporal reasoning.
- Hard negatives markedly increase task difficulty and are recommended for evaluation before deployment.
- The demonstrated success of small-model fine-tuning (4B > 8B) suggests viable cost-effective paths, but real-world robustness needs further validation.
- Uses in healthcare, workplace monitoring, and safety-critical contexts require rigorous trials, privacy protections, and stakeholder governance.
Glossary
- Ablation studies: Controlled experiments that systematically remove or vary components to assess their impact on performance. Example: "we conduct ablation studies."
- Action recognition: The task of identifying actions occurring in videos. Example: "Action recognition has proven to be an evergreen goal of the computer vision community."
- Binary setting: An evaluation format where the model answers a yes/no question about whether a specific action occurs. Example: "we relax the questions into a binary setting"
- Closed models: Proprietary models whose weights and training details are not publicly released. Example: "where the difference between closed and open models is stark"
- Compositional reasoning: The ability to recognize that an action is composed of multiple sub-elements in a specific order. Example: "compositional reasoning (i.e., are all elements of the action present and in the correct order?)."
- Context length: The maximum amount of input (e.g., frames/tokens) a model can process at once. Example: "A discussion of these models' context lengths--which may impact few-shot performance--is available in Appendix \ref{appendix:eval}."
- Coarse-grained labels: Broad action categories that do not distinguish fine details. Example: "predominantly contain coarse-grained labels"
- Distillation: Leveraging outputs from a larger/more capable model to supervise training or labeling for another model. Example: "so distilling directly from even the best-performing VLM is unideal."
- Domain-specific actions: Actions that are defined within and depend on specialized domains (e.g., figure skating, surgery). Example: "we advocate for a returned focus on domain-specific actions."
- Egocentric videos: Videos captured from a first-person viewpoint. Example: "it is restricted to egocentric videos."
- Few-shot learning: Adapting to new tasks or classes using only a few examples at test time. Example: "Inspired by few-shot learning"
- Fine-grained actions: Actions that require distinguishing subtle visual or temporal differences. Example: "the VLM community has focused on fine-grained actions that are not domain-specific"
- Fine-tuning: Additional training of a pre-trained model on a target dataset to adapt it to a specific task. Example: "Fine-tuning a Molmo2-4B model on our data"
- FPS sampling: Selecting video frames at a specified rate measured in frames per second. Example: "fps sampling for Qwen3-VL (2fps), GPT (1fps), and Gemini (1 fps)."
- Hard negatives: Challenging negative examples that closely resemble positive cases but differ in subtle, critical ways. Example: "we create challenging ``hard negatives''"
- Instruction-tuned VLM: A vision-LLM further trained to follow natural-language instructions. Example: "an instruction-tuned VLM,"
- Localizer: A component or model that identifies when relevant actions occur in a video by providing temporal boundaries. Example: "using Gemini 2.5 Flash as a localizer."
- Long-tail: The set of less common domains or actions that have limited data. Example: "for domains in the long-tail, coverage becomes an important factor."
- MLP connector module: A multilayer perceptron used to bridge representations between the vision and language components. Example: "connected to a LLM via a MLP connector module."
- Multiple-choice evaluation setting: An evaluation format where a model chooses the correct action label from several options. Example: "We begin with a multiple-choice evaluation setting"
- Open-weight models: Models whose parameters are publicly available for use and fine-tuning. Example: "the best open-weight 8B VLM attains 45.0\% accuracy"
- Optical flow models: Models that estimate motion between consecutive frames via dense flow fields, often used for action understanding. Example: "and optical flow models \cite{quovadis_kinetics} can be found in Appendix \ref{appendix:traditional_models}."
- Outlier detection: Identifying data points that deviate significantly from others, used here to spot mislabeled or irrelevant clips. Example: "further simplifying this task to an outlier detection problem."
- Post-training: Additional training after initial pre-training, often on specialized data, to improve task-specific performance. Example: "Finally, we explore post-training on domain-specific action data."
- Proprietary models: Non-open models provided by companies, often accessible via APIs but without public weights. Example: "For proprietary models, we use Gemini 3.1 Pro, Gemini 3 Flash, GPT-5.4, and GPT-5."
- Random baseline: A performance reference obtained by guessing, used to contextualize model accuracy. Example: "A category-wise random baseline is the best accuracy attainable by guessing 1 letter for all questions in that category; the overall random baseline is the best accuracy attainable by guessing 1 letter for the entire benchmark."
- Random negatives: Negative examples selected without special difficulty constraints, typically by random sampling. Example: "One approach is to gather ``random negatives'' by randomly sampling different actions within the same domain."
- Taxonomy: A structured, hierarchical organization of concepts—in this case, actions and domains. Example: "We employ a top-down approach to generate our taxonomy of actions."
- Temporal attributes: Time-dependent properties of movements, such as ordering or direction over time. Example: "fixate on fine-grained temporal attributes"
- Temporal boundaries: Start and end times that delineate when an action occurs within a video. Example: "to refine their temporal boundaries."
- Temporal resolution: The fineness of temporal sampling in a video, affecting how well rapid or subtle motions are captured. Example: "models struggle to leverage higher temporal resolution for capturing subtle or rapid motions."
- Uniform sampling: Selecting frames at evenly spaced intervals across a video. Example: "uniform sampling for InternVL3.5-8B (max 48 frames);"
- Video in-context learning: Learning from example videos provided in the prompt at test time to perform new video tasks. Example: "video in-context learning."
- Vision-LLM (VLM): A model jointly processing visual and textual inputs to perform multimodal tasks. Example: "modern vision-LLMs (VLMs) are no longer evaluated on their action recognition capabilities."
- Vision Transformer (ViT): A transformer-based neural architecture applied to images or frames using patch embeddings and self-attention. Example: "consists of a vision transformer (ViT)~\cite{dosovitskiy2021vit}"
- VQA (video question-answer pairs): Paired data where a model answers questions about video content. Example: "We generate 3 video question-answer (VQA) pairs from each clip"
- WhisperX: A tool for extracting accurate, word-level timestamps from audio transcripts. Example: "we extract word-level timestamps using WhisperX~\cite{bain2022whisperx}."
- Zero-shot (0-shot) evaluation: Testing a model on tasks without providing task-specific examples during inference. Example: "Binary 0-shot evaluation results."
Collections
Sign up for free to add this paper to one or more collections.
