VisualClaw: Real-Time Multimodal Agent

• VisualClaw is a self-evolving, multimodal agent that uses hybrid encoding and skill evolution to optimize long-form video analysis in real-time edge settings.
• It employs a three-timescale framework combining on-device frame filtering, prompt-cost control with hot/cold skill injection, and offline skill bank evolution based on failure cases.
• The system improves reasoning and tool-using workflows by adapting its language-level scaffold via episodic memories without fine-tuning the fixed VLM backbone.

VisualClaw is a self-evolving, multimodal agent architecture designed to make long-form video understanding and tool-using workflows both accurate and cost-effective in real-time edge settings. It is built around two complementary principles—hybrid encoding and skill evolution—which allow a fixed, frozen VLM backbone such as Gemini 3 Flash or GPT-5.2 to operate over streaming video with minimal API calls while continuously adapting its language-level reasoning scaffold of skills and memories without updating model weights (Tu et al., 15 Jun 2026). A consolidated technical overview of the separate "ClawMachine" system states that ClawMachine is "also referred to as VisualClaw"; this suggests that the label has been used ambiguously across distinct multimodal systems, even though the 2026 VisualClaw work addresses real-time video-agent deployment rather than referential token grounding (Ma et al., 2024).

1. Problem formulation and architectural scope

VisualClaw is motivated by three deployment gaps in contemporary VLM-based agents. First, dense video streams and long prompts incur high latency and cost. Second, the agent scaffold remains static after deployment. Third, standard video-QA benchmarks do not test whether agents can use visual evidence inside tool-using workspaces. The architecture addresses these gaps by combining on-device frame filtering, prompt-budget control for a growing skill bank, and an offline mechanism that distills reusable reasoning skills from failures (Tu et al., 15 Jun 2026).

The system is described as a three-timescale hybrid-encoding+evolution framework. At the per-frame timescale, a CPU-only cascade gate triages streaming frames into “major,” “minor,” or “skip,” and only major scene changes are uploaded. At the per-question timescale, the skill bank is split into a hot set and a cold catalogue so that prompt cost depends on the selected top-$k$ skills rather than the full bank. At the longer adaptation timescale, batches of failed queries trigger an offline evolver that updates the skill bank using failures and retrieved memories.

A central architectural point is that adaptation occurs entirely at the scaffold level. VisualClaw stores memories, retrieves high-confidence exemplars, proposes skill-bank updates, and prunes low-value or near-duplicate skills, but it does not fine-tune the VLM backbone. This distinguishes self-evolution of the agent policy from weight-space learning. A common misconception is that continual improvement in such systems requires continual parameter updates; VisualClaw is explicitly framed as improving over time without any weight fine-tuning.

2. Hybrid encoding: cascaded visual filtering and hot/cold skill injection

Hybrid encoding is the cost-control mechanism. In the motivating setting, raw video at 1 fps for a 1-hour session yields 3 600 frames and millions of tokens, making naïve upload prohibitively expensive. VisualClaw therefore performs aggressive on-device filtering before any network traffic and separately constrains prompt growth as the skill bank expands (Tu et al., 15 Jun 2026).

The visual side uses a three-stage cascaded gate $G$ over incoming frames $f_t$:

1. A perceptual-hash filter drops $f_t$ if $\mathrm{Hamming}(\mathrm{dHash}(f_t), h)\le \delta$ for any recent hash in a rolling buffer, with $\delta\approx 6$.
2. A lightweight 128-D CPU encoder maps each retained frame to $e_t\in\mathbb{R}^{128}$ using HSV histogram + luminance + edge density.
3. An adaptive change gate compares $e_t$ to a rolling reference $r_t$ using cosine similarity,

$\cos(e_t,r_t)=\frac{e_t\cdot r_t}{\|e_t\|\|r_t\|}.$

The verdict is then

$G$0

with

$G$1

Only frames with $G$2 enter the cloud as the keyframe set $G$3.

The prompt side uses hot/cold top-$G$4 skill injection. Let $G$5 be the current skill bank and $G$6 the incoming question. Both are embedded via a sentence-transformer, similarities $G$7 are computed, and

$G$8

The prompt includes the full text of each $G$9, plus a compact catalogue for each skill in $f_t$0 consisting of “SkillName: one-line description.” The stated effect is that prompt-token cost is capped at $f_t$1 rather than $f_t$2.

In the reported experimental setting, video is sampled at 1 fps, forwarded keyframes are capped at $f_t$3, and cascade thresholds are $f_t$4 and $f_t$5. The empirical claim that cascade-selected keyframes outperform uniform sampling at matched $f_t$6 suggests that scene-change-aware selection preserves more task-relevant signal than evenly spaced frames.

3. Skill evolution, memory integration, and bank hygiene

Skill evolution is VisualClaw’s mechanism for post-deployment adaptation. On every correctly-answered query $f_t$7, the system stores the pair plus context as an embedding in an episodic memory store $f_t$8. When the failure batch reaches $f_t$9, the evolver retrieves the top-$f_t$0 relevant memories

$f_t$1

via $f_t$2 and then proposes skill-bank updates $f_t$3 (Tu et al., 15 Jun 2026).

Two evolver prompting modes are compared. In Direct Concatenation (“Cat.”),

$f_t$4

In Guided Evidence (“Guide”),

$f_t$5

The base prompt $f_t$6 instructs the evolver to “synthesize a set of new, generalizable reasoning skills,” while $f_t$7 tells it to “use these retrieved examples to abstract reusable patterns while avoiding scenario-specific details.” The distinction is operationally important: one mode supplies memories as additional raw context, and the other foregrounds abstraction and de-emphasizes scenario-specific detail.

Bank hygiene is enforced by two filters. F1 rejects any new skill $f_t$8 whose token-Jaccard overlap on names exceeds a threshold,

$f_t$9

with $\mathrm{Hamming}(\mathrm{dHash}(f_t), h)\le \delta$0 given as an example. F2 tracks per-skill hit-rate $\mathrm{Hamming}(\mathrm{dHash}(f_t), h)\le \delta$1 and periodically prunes skills with $\mathrm{Hamming}(\mathrm{dHash}(f_t), h)\le \delta$2. These filters are meant to prevent bank bloat, near-duplicates, and low-utility entries.

The design implies a division between transient experience and distilled procedure. Episodic memory stores successful cases, whereas the skill bank accumulates reusable reasoning patterns. A plausible implication is that VisualClaw’s adaptation is closer to externalized proceduralization than to conventional online learning: the system records exemplars, abstracts them into language-level skills, and reuses those abstractions in future prompts.

4. Video-QA benchmarks, baselines, and quantitative results

VisualClaw is evaluated on four video-QA benchmarks with frozen Gemini 3 Flash and GPT-5.2 via API. Accuracy and cost are the primary metrics, and the ablation grid includes Plain (no skills, uniform-8), Seed (initial bank only), +Evolve (no memory, only skill evolver on failures), +SkillMemCat (memory concatenation at answer-time), FullEvo (Cat./Guide), and Uniform-8 Plain (Tu et al., 15 Jun 2026).

Benchmark	Description	Size
EgoSchema	egocentric, 500 clips, avg 3 min, 5-way MC	500 clips
EgoPlan-Bench	egocentric planning from single frame	923 instances
Video-MME long	12 task types, 900 long clips, 30+ min	900 long clips
NextQA	1000 YouTube clips, ∼30 s, causal/temporal reasoning	1000 clips

The seed skill bank is $\mathrm{Hamming}(\mathrm{dHash}(f_t), h)\le \delta$3, and the evolution trigger is $\mathrm{Hamming}(\mathrm{dHash}(f_t), h)\le \delta$4 failures. On Gemini 3 Flash, the streaming cascade +FullEvo(Guide) configuration reports an average $\mathrm{Hamming}(\mathrm{dHash}(f_t), h)\le \delta$5Accuracy versus Plain cascade of +3.85 pp and a peak +15.80 pp on EgoSchema. Against Uniform-8 Plain, the same setting boosts performance by +3.50…+13.00 pp across benchmarks. At matched $\mathrm{Hamming}(\mathrm{dHash}(f_t), h)\le \delta$6, cascade-fill exceeds Uniform-8 by +1.80 pp on NextQA and +3.58 pp on EgoPlan-Bench, which is presented as confirmation that scene-change keyframes carry more signal than uniform sampling.

The cost results are equally central to the system definition. On Gemini, the framework reduces API spend by –98.1 % versus full-frame @1 fps, from $\mathrm{Hamming}(\mathrm{dHash}(f_t), h)\le \delta$710.51 total, and by –25.9 % versus Uniform-8+FullEvo. The abstract summarizes this as an average –98% versus full-frame upload and –25.9% over the offline uniform 8 frame baseline, while boosting accuracy in most settings.

The discussion further reports capability-conditional gains: weaker backbones like Gemini 3 Flash benefit more (+15.8 pp) than stronger ones (GPT-5.2, +4.0 pp), yet both improve from FullEvo. The stated interpretation is cross-VLM transfer of distilled reasoning patterns.

5. VisualClawArena and multimodal agentic evaluation

To address the claim that standard video-QA benchmarks do not test whether agents can use visual evidence inside tool-using workspaces, the work introduces VisualClawArena, a 200-scenario multimodal agentic benchmark built through a strict five-stage pipeline (Tu et al., 15 Jun 2026). The benchmark contains 3 106 steps, and each scenario combines a short clip with a workspace consisting of documents, dynamic updates, and executable checks.

VisualClawArena shifts the evaluation target from single-shot answering to stateful tool use with grounded evidence. This matters because the agent must not only inspect video but also operate within a workspace whose contents can change and whose outputs can be checked through execution. The benchmark therefore probes whether visual evidence remains usable after it enters a broader agent loop.

The reported backends are Codex (GPT-5.5) and Claude Code (Sonnet 4.6) via a staged-workspace bridge. Results are given in macro scenario accuracy. For Codex, VisualClaw(Cat.) reaches 54.27% versus no-evo 51.35% (+2.92 pp) and versus Uniform-8 50.25% (+4.02 pp). For Claude Code, VisualClaw(Cat.) reaches 52.16% versus no-evo 49.00% (+3.16 pp) and versus Uniform-8 43.99% (+8.17 pp). For Claude Code, cascade versus uniform-8 yields –9.5 % total cost.

These results indicate that the same scaffold-level ideas used for video-QA—selective keyframing, skill-bank retrieval, and offline evolution—can carry over to computer-use agent backends. A plausible implication is that VisualClaw is not restricted to answer generation; it functions as a coordination layer between visual evidence, textual procedures, and executable workspace actions.

6. Edge deployment, limitations, and nomenclatural context

VisualClaw is explicitly positioned for edge applications. The on-device cascade runs in <10 µs/frame on CPU, rejects ∼98 % of frames before any network traffic, and reduces a 1-hour streaming session from ∼3 600 API uploads to only 5–20 calls. Per-question latency remains dominated by round-trip API time (100 ms+), so the cascade overhead is described as negligible (Tu et al., 15 Jun 2026).

The paper also records several caveats. Reward hacking can arise when bias in auto-scored failures drifts the bank; the stated mitigations are confidence gating and the F1/F2 filters, but human audits are recommended. VLM-specific skill drift can occur when format-enforcement skills evolved on one backbone regress performance on another, so per-VLM bank tuning or tighter Jaccard dedup may be needed. Dual-use concerns are also explicit: the same cascade that makes wearable assistants viable could reduce costs of covert surveillance, and platform-level governance is advised.

Future directions include adaptive threshold learning for the cascade, meta-parameter tuning of $\mathrm{Hamming}(\mathrm{dHash}(f_t), h)\le \delta$8 in hot/cold injection per backbone, and richer tool-use primitives in the evolver loop. Within the frame of the paper, these are extensions of the same scaffold-level philosophy rather than departures from it.

The name "VisualClaw" also requires contextualization. A consolidated technical overview of "ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension" states that ClawMachine is "also referred to as VisualClaw" (Ma et al., 2024). ClawMachine, however, centers on visual token collectives, a joint vision-language vocabulary of size 48,384, and a decoder-only LLaMA-2–derived transformer of size 7B (LaVIT-7B) for referring and grounding, whereas the 2026 VisualClaw work centers on streaming-video triage, scaffold evolution, and multimodal agentic workspaces. This suggests that "VisualClaw" should be read with attention to publication context: in current usage it denotes a real-time, personalized agent for the physical world, but related literature contains a distinct referential-comprehension system under overlapping terminology.