RepWAM: Representation Visual-Action Tokenizer

• Representation Visual-Action Tokenizer is a representation-centric latent interface that maps visual inputs into semantically aligned visual and action tokens, prioritizing control-relevant features over pixel fidelity.
• It employs a two-stage approach with RepViTok, first extracting semantic visual tokens via a ViT-style autoencoder with frozen foundation model alignment, then modeling transitions using latent action tokenization.
• Empirical evaluations demonstrate improved performance in robotic tasks, with enhanced success rates and coherent scene evolution compared to traditional reconstruction-focused video tokenizers.

Searching arXiv for the cited papers and closely related tokenizer work to ground the article in current literature. A representation visual-action tokenizer is a representation-centric latent interface that maps visual inputs into aligned visual and latent action tokens so that future visual states and the actions connecting them can be jointly modeled under language instructions. In RepWAM, this idea is introduced as an alternative to the reconstruction-oriented video tokenizers commonly inherited from pretrained video generation models: instead of allocating most capacity to pixel fidelity, the tokenizer is designed to preserve semantic structure relevant to manipulation and to encode action as a transition in the same latent space (Wang et al., 11 Jun 2026).

1. Conceptual definition and motivation

The defining motivation is the mismatch between reconstruction-oriented tokenization and control-oriented world modeling. Existing world action models typically inherit video tokenizers optimized for pixel reconstruction. RepWAM argues that this design is poorly matched to instruction-following dynamics, because reconstruction-heavy tokenizers spend capacity on background texture, lighting, and other low-level appearance details, while manipulation depends more directly on object identity, spatial layout, affordances, and contact cues. It further argues that conventional world action models keep visual latents and robot actions in disjoint spaces, forcing an inverse dynamics model to bridge that gap repeatedly (Wang et al., 11 Jun 2026).

Within this formulation, a representation visual-action tokenizer is not merely a compressor. It is intended to produce a semantic visual-action latent space that is action-aware and transferable to robot control. The visual component should preserve the factors of variation that matter for future prediction under language instructions, while the action component should encode the transition between latent visual states rather than a robot-specific motor command. A plausible implication is that tokenization becomes part of the control representation itself, rather than a preprocessing layer for a separate dynamics model.

This design goal differs from image or video tokenizers whose primary criterion is reconstruction quality. RepWAM’s central position is that high-fidelity visual reconstruction is not the right end goal for robot control, and that the relevant bottleneck is whether the latent space preserves manipulation-relevant semantics and action structure (Wang et al., 11 Jun 2026).

2. Semantic visual tokenization in RepViTok

RepWAM’s tokenizer, called RepViTok, is organized in two stages. The first stage is a semantic visual tokenizer implemented as a ViT-style autoencoder. Input observations $o_{1:T}$ are patchified such that the first frame $o_1$ is divided into $16\times 16$ patches and later frames $o_{2:T}$ into $4\times 16\times 16$ tubelets. These tokens are processed by an encoder $E_\theta$ with temporal-causal masking across frames and full spatial attention within each frame; a projection layer and layer norm then produce video latents $z$, while a symmetric decoder $D_\theta$ reconstructs pixels (Wang et al., 11 Jun 2026).

The critical departure from reconstruction-first tokenizers is the addition of semantic alignment to a frozen visual foundation model $G$, termed the Perception Encoder. The visual objective is

$$\mathcal{L}_{\mathrm{vis}}=\mathcal{L}_{\mathrm{rec}}+\lambda_{\mathrm{align}}\mathcal{L}_{\mathrm{align}},$$

where the reconstruction term combines $o_1$0, perceptual, and adversarial losses, and the alignment term matches temporally averaged latent features to the temporally averaged output of $o_1$1. In the paper’s interpretation, this is the representation-centric step: reconstruction quality is retained, but the latent space is explicitly biased toward the semantics encoded by a frozen foundation model (Wang et al., 11 Jun 2026).

This architecture defines the visual half of a representation visual-action tokenizer. The goal is not to eliminate reconstruction, but to subordinate it to a broader requirement: the learned visual tokens must remain useful for downstream world modeling, instruction conditioning, and later action induction. A common misunderstanding is to treat this as standard autoencoding with an auxiliary loss. RepWAM frames it more strongly: semantic alignment is the mechanism by which the visual latent space becomes appropriate for manipulation rather than video generation alone.

3. Latent action tokenization as semantic transition modeling

The second stage of RepViTok is a latent action tokenizer trained after the visual tokenizer is frozen. It operates on pairs of latent visual frames $o_1$2 and couples an inverse dynamics model $o_1$3 with a forward dynamics model $o_1$4: $o_1$5 Here $o_1$6 is the latent action token, $o_1$7 is a soft transport operator over spatial tokens, and $o_1$8 is a residual update (Wang et al., 11 Jun 2026).

The transport formulation is central. The paper characterizes $o_1$9 as moving information across spatial tokens in a manner analogous to optical-flow-style transport in semantic token space, while $16\times 16$0 captures changes not explained by transport alone. Because the action token is defined over visual state transitions, it is described as less tied to a specific robot embodiment. RepWAM therefore treats the action token not as a generic code extracted from pixel change, but as a compact semantic transition variable between aligned visual states (Wang et al., 11 Jun 2026).

Training uses a forward prediction loss and a backward consistency loss: $16\times 16$1 This establishes a two-stage interpretation of action tokenization. First, semantic visual states are learned. Second, actions are induced as structured transitions inside that semantic state space. The paper’s ablations later show that this separation is materially important: latent actions are useful, but joint prediction that entangles state prediction and action supervision degrades generation quality relative to the two-stage procedure (Wang et al., 11 Jun 2026).

4. Use inside a world action model

RepWAM places the tokenizer inside a causal diffusion transformer that jointly models visual tokens and latent action tokens. Given a language instruction $16\times 16$2, observations, and latent actions, the model groups tokens into chunks,

$16\times 16$3

and forms a sequence containing text embeddings from a frozen text encoder, the initial visual context, and multiple visual-action chunks. A block-causal mask ensures that each chunk attends only to past context rather than future chunks (Wang et al., 11 Jun 2026).

The world action model is trained by flow matching. For each chunk, Gaussian noise is interpolated with the target visual-action chunk, and the transformer predicts separate velocities for visual tokens and latent actions. The stated objective jointly supervises future semantic visual states and the latent actions that explain those transitions under language conditioning. This makes the tokenizer the substrate for both prediction and control rather than a frozen perceptual front end (Wang et al., 11 Jun 2026).

Pretraining uses AgiBot with about 100G video-action latent tokens, with latent action components produced by RepViTok. For finetuning, the model uses a mixed real-robot corpus from AgiBot, RoboMIND, RoboCOIN, and InternA1 totaling about 300G tokens, where each demonstration includes continuous motor commands aligned with visual trajectories. Real-robot evaluation is conducted on a Franka dual-arm robot for three tasks: pick fruits and put them into the plate, push the drawer and put the building block into it, and insert the test tube into the rack. For each task, finetuning uses 50 real demos, 500 steps, learning rate $16\times 16$4, and sequence length 150K (Wang et al., 11 Jun 2026).

A notable implication is that the latent actions remain semantic and transferable, but still require a robot-specific decoder or action expert to map them to executable motor commands. The tokenizer therefore does not eliminate embodiment adaptation; it restructures where embodiment specificity enters the system.

5. Empirical behavior, ablations, and technical claims

The empirical results in RepWAM are used to argue that semantic visual-action tokenization outperforms reconstruction-oriented tokenization for world action modeling. On the three Franka tasks, both model sizes reach 60% on pick fruit; on push drawer, the $16\times 16$5B model reaches 50% and the $16\times 16$6B model 80%; on insert tube, the $16\times 16$7B model reaches 30% and the $16\times 16$8B model 60%. The largest gains appear on long-horizon and fine-grained tasks, which the paper interprets as evidence that semantic tokenization helps maintain coherent scene evolution across multiple steps (Wang et al., 11 Jun 2026).

On the 50-task RoboTwin 2.0 simulation benchmark, the $16\times 16$9B model achieves 89.3 on Easy and 88.4 on Hard. The paper states that this beats $o_{2:T}$0 and Motus and is competitive with or close to Lingbot-VA. It also notes that Lingbot-VA likely still benefits from WAN video-generation pretraining, whereas RepWAM is trained from scratch. This is presented as evidence that the tokenizer itself is strong even without a pretrained generative video backbone (Wang et al., 11 Jun 2026).

The tokenizer ablations are especially diagnostic. RepViTok is compared with WAN2.2 VAE and ViTok, and is reported to obtain the best gFVD, the best OLS, and the best real-robot success. On the seen set, OLS improves from 13.68 with WAN2.2 VAE to 18.82 with RepViTok; on the unseen set, from 11.21 to 14.15; PickFruit success improves from 20% to 30%. In a controlled backbone comparison with the same $o_{2:T}$1B world action model, replacing WAN2.2 VAE with RepViTok changes results from 78.0 / 76.0 to 86.6 / 83.1. The intended conclusion is explicit: better reconstruction does not automatically imply better control (Wang et al., 11 Jun 2026).

The latent-action ablation further clarifies the representation design. The paper compares a model without latent actions, a Joint Pred variant, and the Two Stages method. Joint Pred is said to entangle state prediction and action supervision, thereby hurting generation quality. Two Stages is best on every metric: for seen gFVD, 61.01 without latent actions, 94.25 for Joint Pred, and 48.23 for Two Stages; for PickFruit, 30%, 20%, and 50%, respectively. The paper also reports that RepViTok latent actions focus more sharply on manipulation-relevant changes than LAPA and are easier for an inverse dynamics model to map into actual robot actions. Finally, the best average success rate occurs at video CFG $o_{2:T}$2, suggesting that extra classifier-free-guidance extrapolation is unnecessary when the latent space is already well aligned and semantically grounded (Wang et al., 11 Jun 2026).

6. Relation to broader visual tokenization research

Representation visual-action tokenization emerges from a broader rethinking of discrete visual bottlenecks. UniTok argues that unified tokenizers underperform not because reconstruction and semantic supervision inherently conflict, but because discrete visual tokens have insufficient representational capacity; it addresses that bottleneck with multi-codebook quantization and attention-based factorization, reaching 0.38 rFID and 78.6% zero-shot ImageNet accuracy in its strongest variant while also supporting native visual generation in multimodal LLMs (Ma et al., 27 Feb 2025). DualToken reaches a different conclusion for image tokenization: it treats semantic–perceptual conflict as genuine at the level of a single codebook and resolves it by introducing separate semantic and perceptual vocabularies, reporting zero-shot ImageNet accuracy 81.6 together with rFID 0.54, PSNR 23.56, and SSIM 0.742 (Song et al., 18 Mar 2025).

V2Flow shifts the bottleneck from codebook design to vocabulary alignment with pretrained LLMs. It maps each visual token to a soft categorical distribution over the LLM vocabulary, compresses images into a compact 1D sequence, and reconstructs with a masked autoregressive rectified-flow decoder; on ImageNet-1K reconstruction at $o_{2:T}$3, it reports PSNR 22.37, SSIM 0.65, and LPIPS 0.20, and it is integrated with LLaMA2-7B for autoregressive text-conditioned image generation (Zhang et al., 10 Mar 2025). MAGVIT-v2, by contrast, emphasizes tokenizer quality as the decisive factor for visual generation and video understanding, replacing conventional vector quantization with lookup-free quantization and showing that the same discrete vocabulary can support both generation and action recognition; its reported action-recognition results include 62.40 on SSv2, 75.34 on K400, and 77.93 on K600 when tokens are used as inputs (Yu et al., 2023).

The closest analogue outside robotics is Unified Driving Tokens, which proposes a representation-guided and geometry-enhanced tokenizer for autonomous driving. It aligns its discrete bottleneck to a frozen DINO feature space, preserves appearance with RGB reconstruction plus perceptual and adversarial losses, adds adjacent-frame depth and relative-pose supervision, and stabilizes joint objectives with multi-codebook quantization. On NAVSIM, it reports best rFID 4.15, PSNR 26.51, SSIM 0.774, and a planning score of PDMS 91.8 with a frozen-token lightweight planner (Yao et al., 1 Jun 2026).

Taken together, these works suggest a common trajectory. Visual tokenizers are increasingly treated as representation layers rather than compressors, and the optimization target is shifting from pixel fidelity alone toward semantics, geometry, latent action structure, or compatibility with downstream autoregressive models. Representation visual-action tokenization is the robotics-specific extension of that trend: it preserves semantic visual structure, encodes actions as transitions in the same latent geometry, and uses that shared discrete space as the basis for world modeling and closed-loop control (Wang et al., 11 Jun 2026).