Watch & Learn (W&L) Methods Overview

• Watch & Learn is a paradigm where systems learn by observing behaviors rather than relying on direct supervision, applicable in domains such as autonomous driving, UI automation, and scientific modeling.
• It employs techniques like waypoint inference, inverse dynamics, and multimodal guidance to transform observational data into actionable instructions.
• The framework has improved task performance across benchmarks while highlighting challenges in noise handling, occlusion, and evaluation contamination.

“Watch & Learn” (W&L) is not a single standardized method in the arXiv literature. Rather, it is a recurring research label for methods, benchmarks, and conceptual programs in which supervision is derived from observation: watching other drivers, online tutorial videos, equilibrium configurations, training checkpoints, human users, or interaction outcomes. In current usage, two especially direct instantiations are autonomous driving by observing surrounding vehicles—formalized as “Learning by Watching” (LbW)—and computer-use learning from online screencasts, where raw videos are converted into executable user-interface trajectories (Zhang et al., 2021, Song et al., 6 Oct 2025).

1. Term, scope, and recurring meanings

Across fields, “Watch & Learn” and closely related titles have been used for substantially different technical objects. In some papers it is the formal method name, as in “Watch and Learn: Learning to Use Computers from Online Videos” (Song et al., 6 Oct 2025), “Watch and Learn: Semi-Supervised Learning of Object Detectors from Videos” (Misra et al., 2015), and “Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision” (Zhong et al., 2020). In others it is a title-level framing while the operative method has another name, such as distribution-consistent learning (DCL) in statistical physics (Sprague et al., 2020) or Learning by Watching (LbW) in autonomous driving (Zhang et al., 2021).

The phrase also spans different epistemic regimes. In some cases, the system learns from observing external agents or videos; in others, it learns from its own revealed outcomes or from inspection of optimization trajectories. “Watch and Learn: Optimizing from Revealed Preferences Feedback” studies Stackelberg optimization from observed follower responses rather than known utility functions (Roth et al., 2015). “Watch an AI Weather Model Learn (and Unlearn) Tropical Cyclones” uses checkpoint-wise analysis to inspect how a model acquires and later loses storm-specific skill (Baiman et al., 20 Mar 2026). This breadth indicates that W&L functions less as a domain-specific algorithmic family than as a recurrent paradigm of indirect supervision, weak observation, or observation-driven adaptation.

A further distinction runs between learning from observation and building systems that observe humans in order to guide them. “Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake?” introduces Watch, Talk and Guide (WTaG) as a benchmark for multimodal situated guidance rather than for passive imitation (Bao et al., 2023). “Tutorly: Turning Programming Videos Into Apprenticeship Learning Environments with LLMs” similarly turns recorded instructional content into an adaptive, learner-facing environment rather than treating the video merely as training data (Li et al., 2024).

2. Autonomous driving: Learning by Watching

The clearest autonomous-driving realization of W&L is “Learning by Watching” (Zhang et al., 2021). The central problem is a limitation of standard imitation learning and behavioral cloning: conventional methods assume direct access to an instrumented ego vehicle’s states and expert actions, typically written as

$\tau^{expert}=\{(s_t,a_t^*)\}_{t=1}^{T}.$

LbW replaces this with an estimated non-ego trajectory

$\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$

because watched vehicles are observed only indirectly from the ego viewpoint and their agent-centric state and low-level controls are not directly measurable (Zhang et al., 2021).

The framework has two core steps: transform the ego vehicle’s observations into the watched vehicle’s point of view, and infer the watched vehicle’s expert actions. The method operates on a bird’s-eye-view representation with visibility modeling. The state is

$s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$

where $\mathbf B_t$ is a $7$-channel BEV tensor, $\mathbf M_t$ is a visibility map, $v_t$ is speed, and $c_t$ is a high-level navigation command. Rather than predicting steering, throttle, and brake directly, the policy is decomposed as

$\pi_\theta(s)=g(f_\theta(s)),$

where $f_\theta$ predicts future waypoints and $\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$0 is a fixed PID controller mapping waypoints to control. This decomposition is explicitly motivated by the claim that direct inference of low-level controls from observation alone is too ill-posed, whereas waypoints are recoverable from tracked motion.

Action inference is therefore waypoint inference. Future tracked motion yields

$\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$1

and training uses a waypoint behavior-cloning objective

$\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$2

To reduce pseudo-label noise, the paper introduces refurbishment: $\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$3 where an ego-only baseline model pulls noisy watched labels toward trajectories explainable by the estimated watched-agent state. The training pipeline first collects ego demonstrations $\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$4, then extracts additional LbW pseudo-demonstrations $\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$5 for observed surrounding vehicles, and finally trains from scratch on the union $\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$6.

Experiments are conducted in CARLA 0.9.9 on the original benchmark, NoCrash, and a Town 3 adaptation setting. The headline original-benchmark results are 92% success for LbW + Visibility (Late) with only 30 minutes of total driving data and 82% success for plain LbW with only 10 minutes. On NC-Regular, an LbW agent trained with only 10 minutes reaches 52% success, surpassing an ego-only baseline trained with one hour at 46%. In Town 3 adaptation with only 10 minutes of adaptation driving, the baseline reaches 40/40/60 on NC-R/NC-D/OB, LbW reaches 60/60/80, and LbW + Visibility (Late) reaches 100/40/100. The paper also makes the limitations explicit: simulator-only evaluation, reliance on scene abstraction and 3D tracking, persistent occlusion problems, heuristic command inference, and the assumption that watched drivers are experts.

3. Computer-use agents: executable trajectories from online videos

In computer-use learning, W&L is the formal name of a data-generation framework that converts human demonstration videos on the Internet into executable UI trajectories (Song et al., 6 Oct 2025). The key formulation is inverse dynamics: given two consecutive observations $\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$7, predict the action $\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$8 that caused the transition. The recovered trajectory is represented as

$\hat{\tau}^{expert\mhyphen LbW}=\{(\hat{s}_t,\hat{a}_t^*)\}_{t=1}^{T},$9

This choice is presented as simpler and more learnable than generating whole trajectories end to end, and as reducing ad hoc heuristic engineering.

The pipeline has three stages. First, an inverse dynamics model (IDM) is trained on over 630k triples $s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$0, formed by combining around 500k synthetic transitions from live web interaction with 132k human-annotated transitions from Mind2Web. Second, videos are retrieved and filtered. For inference-time in-context use, Gemini 2.5 Flash generates a concise YouTube search query from the task instruction and initial screen, the top 15 videos are retrieved, and titles are treated as candidate task descriptions. For training-time collection, 69 applications from seven categories are queried. Filtering samples frames at 1 fps, uses Gemini 2.5 Flash to classify frames as categories such as Clean Screencast, Zoomed Screencast, Animated/Transition, Talking Head, Slide/Presentation, or Other, and retains only videos whose average frame score exceeds 0.8. Third, the IDM is run on consecutive frame pairs to assemble executable trajectories.

The action space is deliberately compact: click, scroll, type, wait, and move. For click, move, and type, normalized $s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$1 coordinates are discretized to integers from 0 to 1000 and predicted as classification targets; text for type actions is generated by a language decoder. Architecturally, the IDM uses a SigLIP-2 vision encoder, four Transformer layers, an action classification head, a coordinate head, and a GPT-2 small decoder for text generation. It is trained end-to-end with cross-entropy for action classification, cross-entropy for discretized coordinates, and language modeling loss for text generation. The produced dataset contains 53,125 video-derived trajectories across 69 applications and seven categories, including 12,829 in Programming, 8,691 in Productivity, and 7,948 in Design.

On OSWorld-Verified, W&L trajectories improve both in-context learning and supervised fine-tuning. In-context, Gemini 2.5 Flash improves from 19.0% to 22.0%, OpenAI o3 from 21.8% to 24.3%, Claude 4 Sonnet from 43.9% to 45.5%, and Jedi from 50.6% to 52.8%. In supervised training, Qwen 2.5-VL 7B rises from 1.9% to 13.0%, while UI-TARS-7B improves from 27.3% to 31.1%. A central justification is label fidelity: on held-out Mind2Web, W&L’s IDM reaches 91.6% action accuracy and 96.4% action-type accuracy, compared with 82.7% and 88.9% for TongUI, and 72.8% and 81.4% for Gemini 2.5 Flash as labelers. The paper also notes limitations: the five-action taxonomy excludes drag-and-drop, text decoding remains weaker than click and scroll, retrieval is whole-task rather than subtask-granular, and coverage is biased toward applications with abundant tutorial videos.

4. Interactive guidance, tutoring, and demonstration-conditioned adaptation

A different W&L line treats watching as the front end of interactive adaptation rather than passive data extraction. “Watch, Try, Learn: Meta-Learning from Demonstrations and Reward” formulates a two-stage few-shot control procedure for tasks $s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$2 in which the learner first conditions on demonstrations

$s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$3

and then improves using rewarded trial trajectories

$s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$4

The method trains a trial policy $s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$5 and a re-trial policy $s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$6 with supervised meta-imitation objectives over held-out demonstrations rather than direct policy-gradient RL. On the gripper benchmark, success rises from $s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$7 for meta-imitation learning to $s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$8 for WTL with one trial, while BC + SAC requires roughly 2500 trials to approach $s_t=[\mathbf B_t,\mathbf M_t,v_t,c_t],$9 (Zhou et al., 2019).

WTaG addresses a complementary problem: whether a system can watch a human user in real time and provide situated task guidance (Bao et al., 2023). The benchmark comprises nearly 10 hours of egocentric cooking interaction across 56 task sessions, 17 user subjects, and 3 instructor subjects, with 4,233 dialogue utterances and annotations for recipe steps, user intents, mistake types, instructor intents, and instruction subtypes. It formalizes two tasks, User and Environment Understanding and Instructor Decision Making, with query points triggered when the user speaks, the instructor speaks, or no one has spoken for 10 seconds. Zero-shot foundation-model baselines perform above random on some subtasks such as user intention prediction and step detection, but “when to talk” remains around chance, BLIP-2 scene descriptions are truthful only about 25–30% of the time, and the models tend to over-instruct.

Tutorly applies the watch-and-learn idea to programming videos by converting them into an apprenticeship-style environment inside JupyterLab (Li et al., 2024). The system segments transcripts by learning goals, extracts knowledge summaries, chooses cognitive apprenticeship moves—Modeling, Coaching, Scaffolding, Articulation, Reflection, and Exploration—and drives an LLM-based mentor with a student model. The learner model uses a Bayesian Knowledge Tracing update

$\mathbf B_t$0

parameterized by $\mathbf B_t$1, $\mathbf B_t$2, and $\mathbf B_t$3, and it affects both goal prioritization and pedagogy selection. In a within-subject study with 16 participants learning exploratory data analysis, performance improves from 61.9% to 76.6%. Reported cognitive load is low on extraneous load and high on germane load, and usability is high, but the paper also notes limitations in transcript quality, segmentation accuracy, and LLM latency.

5. Physics, weather, and surgery: scientific and domain-specific variants

In statistical physics, “watch and learn” appears as a title-level framing for DCL, a two-stage unsupervised method that observes equilibrium configurations at a single temperature and infers a transferable energy model (Sprague et al., 2020). The first stage uses an autoregressive RNN to learn

$\mathbf B_t$4

which is then converted into energy differences by

$\mathbf B_t$5

The second stage trains an extensive deep neural network (EDNN) on RNN-derived energies to impose locality, extensivity, and uniformity of physical law. Training uses a single observation temperature, typically $\mathbf B_t$6, yet the learned model extrapolates across temperatures and phases, transfers from $\mathbf B_t$7 to $\mathbf B_t$8 Ising systems, and applies to Potts and Edwards–Anderson spin-glass models. On $\mathbf B_t$9 Ising, the EDNN reduces RMSE from $7$0 for the RNN energy model to $7$1.

In surgical video understanding, W&L denotes a two-stage video-language framework trained on expert-commented educational videos (Gastager et al., 14 Mar 2025). Stage 1 is a modified VALOR$7$2 model with VideoSwin$7$3, BERT$7$4, and multimodal objectives for alignment, generative captioning, and masked language modeling: $7$5 Stage 2 uses a temporal model—MS-TCN++, ASFormer, or ASMamba—to aggregate clip representations over full procedures. A large YouTube cataract dataset is built from 2,933 videos, 149,939 clips, and 2,247,750 words after filtering. Downstream labels from datasets such as CATARACTS, Cataract-101, and CholecT50 are projected into language templates and adapted with LoRA. The best CATARACTS result, V-YT-LoRA-CAT + ASM, reaches accuracy $7$6 and F1 $7$7; the best Cataract-101 result reaches accuracy $7$8 and F1 $7$9. The paper also reports zero-shot gains over OphCLIP and proposes a first practical surgical dense video captioning pipeline.

The weather-model paper uses “watch-and-learn” in a different sense: watching the model itself learn (Baiman et al., 20 Mar 2026). Every checkpoint of an SFNO trained on ERA5 is saved across 70 epochs plus 20 epochs of 2-step fine-tuning, yielding 90 checkpoints. Evaluation over 106 tropical cyclones defines storm-specific learning trajectories using a 1200 km storm-relative domain and intensity/location errors based on minimum mean sea-level pressure. K-means clustering of checkpoint-wise intensity trajectories reveals three storm groups, including a Cluster 3 of 23 storms for which intensity skill improves until roughly checkpoint 30 and then deteriorates. These storms occur in anomalously moist environments, and the paper argues that the model may be unlearning the relationship between moisture and TC intensity rather than failing to predict the moisture anomaly itself.

6. Alignment, grounding, and the limits of observation

Several W&L papers converge on the problem that observation is noisy, weakly aligned, or shortcut-prone. “Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision” learns sentence–video alignment from subtitle-timed web video rather than manual labels (Zhong et al., 2020). The full WAL-att-adv model uses sentence-guided attention and an adversarial routing mechanism with a discrete latent variable $\mathbf M_t$0 to decide whether a pair should contribute to the main language-visual correspondence loss or to an adversarial branch. On the ApartmenTour dataset of 2,906 YouTube videos and around 500k weak sentence–video pairs, WAL-att-adv reaches 30.1 video-search mAP and 30.0 sentence-search mAP, outperforming SCAN and PVSE on the reported retrieval benchmarks.

A related fault line is evaluation contamination. “Watch Before You Answer” shows that widely used long-video benchmarks contain large fractions of text-only answerable questions, with reported text-only accuracies of 58.2 on VideoMME, 61.1 on VideoMMMU, and 63.4 on MMVU for Gemini-3.1-Pro (Zhang et al., 6 Apr 2026). In the Video-R1-260K post-training dataset, GPT-5-mini labels 81,361 of 263,071 samples as text-answerable, retaining 181,710 visually grounded questions, or 69.1% of the original set. Using only this curated visually grounded subset improves Full Avg by up to 6.2 points relative to training on the full dataset while using less data. The paper’s central claim is therefore not merely that shortcuts distort benchmarks, but that post-training itself can be undercut when reward and supervision do not require actual visual grounding.

The broader survey “Watch, Remember, Reason” formalizes this issue by decomposing video understanding into

$\mathbf M_t$1

followed by output generation from perceptual representations, memory states, and reasoning traces (Meng et al., 5 Jun 2026). The survey’s core thesis is that long-form video understanding requires not only watching but also remembering and reasoning under limited context and compute. This suggests a general lesson across W&L variants: observation alone rarely suffices. The recurrent technical responses are explicit structure—BEV plus visibility maps in driving, inverse dynamics labels in CUAs, trajectory-conditioned retry in control, DSL-constrained mentoring in tutoring, adversarial routing in weak video-language alignment, and visual-grounding filters in post-training. Across these literatures, W&L denotes a family of solutions to the same underlying difficulty: how to extract actionable supervision when the latent state, action, or meaning is only partially visible.