FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation

Abstract: Vision-language-action (VLA) models have achieved impressive generalization in robotic manipulation, and recent memory-augmented VLAs have relaxed the Markovian assumption by conditioning on past images or language summaries. Vision-based memory approaches address this by conditioning on sampled past image frames, but they are computationally expensive and fundamentally limited when temporal events are visually ambiguous, e.g., pushing a button multiple times with small movements. We propose FM-VLA, a VLA model with force-based memory, enabling temporal context reasoning for non-Markovian, contact-rich manipulation. We encode force histories into compact force memory tokens with a variational autoencoder (VAE) pretrained with force time series reconstruction. By projecting force latent representations and short state history as additional conditioning tokens to the action expert module, we enable VLAs to leverage accumulated contact event history to guide manipulation. We evaluate FM-VLA on three memory-dependent tasks, including finding a hidden block, pressing a button, and wiping a dish for a specific number of times. Our lightweight force memory achieves over 80% success rate with minimal inference overhead, significantly outperforming baseline approaches. Project page: https://qft-333.github.io/FM-VLA-Page/

• The paper introduces a variational autoencoder that compresses full wrist-wrench histories into eight latent memory tokens, which condition a [?] vision-language-action policy alongside short-window proprioceptive state context.
• FM-VLA achieves 83.3% average success across hidden-object, button-counting, and bowl-wiping tasks, outperforming visual memory at 53.7% and reaching 100% on the cup task.
• The results show that force memory and short-term state context are complementary, while the method adds only 3.3 ms of inference latency but remains limited by fixed token capacity, single-robot evaluation, and uncertain cross-task generalization.

FM-VLA addresses a specific failure mode of memoryless vision-language-action (VLA) policies: contact-rich manipulation tasks whose success depends on accumulated interaction history that is invisible or ambiguous in the visual stream. The paper's central claim is that the wrist wrench signal, rather than past images or language summaries, is the appropriate substrate for long-horizon memory in such tasks, because contact events, force magnitudes, and action counts are directly and unambiguously encoded in force measurements. The proposed system augments $\pi_{0.5}$ with a variational autoencoder (VAE) that compresses the full wrench history into a small set of latent tokens, which condition the flow-matching action expert alongside a single short-window proprioceptive state token. On three memory-dependent bimanual tasks, FM-VLA reaches an 83.3% average success rate versus 53.7% for a visual-memory baseline, while adding only 3 ms of inference latency.

Motivation and positioning

Existing memory-augmented VLAs operate on visual or linguistic tokens. MemoryVLA maintains a perceptual-cognitive memory bank over past observations, and MEM combines a short-term video memory with long-term language summaries. The paper argues that these approaches fail when task-relevant state changes produce negligible visual change — for instance, pressing a button whose mechanical travel is nearly invisible, or wiping a bowl where successive passes look identical. Conversely, force-augmented VLAs such as ForceVLA and TA-VLA condition on a short window of concurrent wrench readings, providing corrective feedback for the immediate motion but no episodic record of prior interactions. FM-VLA is positioned as the first VLA to treat the wrench stream as a long-horizon memory rather than instantaneous conditioning.

The policy is formulated as $\pi(a_t \mid o_t, l, h_t)$, where $h_t$ concatenates two complementary streams: a full-episode 6-axis wrench history from a wrist-mounted F/T sensor, and a short window (~0.9 s) of joint positions and gripper states. The short state stream exists to fix a specific pathology the authors observed: conditioning on force history alone induces repetitive pre-contact behaviors, because the policy lacks awareness of its current motion phase.

Architecture and training

FM-VLA is built on $\pi_{0.5}$, comprising a PaliGemma VLM with a SigLIP vision encoder and a flow-matching action expert. Training proceeds in two stages.

Stage 1: Force-VAE pretraining. A Perceiver-IO encoder/decoder VAE is trained on wrench time-series reconstruction across all tasks jointly, with inverse-frequency task sampling. The objective is a masked reconstruction loss plus a free-bits-regularized KL term, where a per-dimension free-bits floor prevents posterior collapse. Inputs are EMA-smoothed (causal first-order filter, $\alpha = 0.3$) and quantile-normalized. A notable detail is randomized noise pre-padding: because history length leaks episode progress, the model would otherwise shortcut on sequence length; prepending a random-length Gaussian noise prefix (up to 10 s) during training forces the encoder to rely on signal content. The encoder compresses the unbounded history into $K$ latent tokens via cross-attention into learnable latent queries interleaved with latent self-attention.

Stage 2: VLA fine-tuning. The frozen VAE encoder's posterior mean $\mu_f$ is projected through a zero-initialized linear layer into $K$ wrench memory tokens, appended after the noisy-action tokens in the action-expert suffix, followed by a single state token from a zero-initialized linear projection of a flattened 10-frame joint-state window. This placement preserves the RoPE positions of the noisy-action tokens from base-policy pretraining. The VLM, action expert, and both projectors are fine-tuned jointly with the rectified-flow objective of $\pi_{0.5}$, predicting the constant velocity of the straight-line flow path.

Experimental results

Experiments use an AgiBot G1 bimanual humanoid with two 7-DoF arms, 1-DoF grippers, and 6-axis wrist F/T sensors sampled at 100 Hz. Three tasks are designed to be non-Markovian and visually ambiguous: finding a hidden block under two visually identical cups (requiring memory of which cup was inspected), pressing a button exactly $N \in \{1,2,3\}$ times (each press yields a sharp wrench impulse but negligible visual displacement), and wiping a bowl for $\pi(a_t \mid o_t, l, h_t)$0 passes. Training data comprise 200, 350, and 200 teleoperated demonstrations respectively; evaluation uses 18 trials per task with strict success criteria (e.g., exactly $\pi(a_t \mid o_t, l, h_t)$1 audible clicks, no re-inspection of cups).

Method	Cups	Buttons	Wipe	Average
$\pi(a_t \mid o_t, l, h_t)$2 (no history)	72.2	11.1	0.0	27.8
TA-VLA (short force window)	50.0	11.1	5.6	22.2
$\pi(a_t \mid o_t, l, h_t)$3-MEM (visual memory, $\pi(a_t \mid o_t, l, h_t)$4)	77.8	33.3	50.0	53.7
FM-VLA (VAE)	100.0	72.2	77.8	83.3

FM-VLA achieves a perfect 100% on the Cups task and substantially outperforms the visual-memory baseline on the contact-counting tasks (72.2% vs. 33.3% on Buttons). The gap between $\pi(a_t \mid o_t, l, h_t)$5-MEM and FM-VLA on Buttons is the paper's strongest evidence that visual memory is fundamentally mismatched to tasks lacking observable state change. TA-VLA's poor performance (22.2% average) indicates that a short force window cannot substitute for episodic contact memory.

Modality ablations show that neither stream suffices alone: force-only collapses to 25.9% average due to erratic pre-contact motion, while state-only achieves 100% on Cups but fails on contact-counting tasks (11.1% on Buttons). The two streams are complementary — long-term contact events versus short-term motion context.

Architecture ablations support the VAE design. A GRU recurrent encoder degrades to 33.3% average, losing early contact events over the long 100 Hz sequences (5.6% on Wipe), and an end-to-end Q-Former reaches only 57.4%, overfitting to instantaneous force peaks rather than holistic temporal structure. The reconstruction-pretrained VAE latent space, by contrast, is forced to encode macroscopic structure (force magnitudes, onsets, contact counts) in few tokens.

Token capacity is a non-monotonic factor: 8 tokens is optimal, 4 tokens form an informational bottleneck, and — somewhat counterintuitively — 16 and 32 tokens degrade performance. The authors attribute this to distribution shift, since the pretrained $\pi(a_t \mid o_t, l, h_t)$6 action expert observes at most 50 tokens during its original training, so 32 extra force tokens exceed this limit. This is an assumption rather than a demonstrated mechanism; no direct evidence isolating the RoPE/token-budget hypothesis is provided.

Inference efficiency is a practical strength: FM-VLA adds 3.3 ms over the 60.7 ms base policy on an RTX 4090, whereas $\pi(a_t \mid o_t, l, h_t)$7-MEM with $\pi(a_t \mid o_t, l, h_t)$8 frames adds 39.1 ms and with $\pi(a_t \mid o_t, l, h_t)$9 reaches 190 ms. The force-based memory is thus roughly an order of magnitude cheaper than visual memory at the evaluated operating point.

Limitations and open questions

The paper concedes two main limitations. First, the VAE imposes a fixed bottleneck of 8 tokens, which may be inadequate for very long-horizon tasks involving hundreds of contact events; the authors note that hierarchical or adaptive compression may be required, but do not evaluate any such scheme. Second, the VAE is pretrained only on wrench data from the task demonstration dataset; whether the latent space generalizes across embodiments or benefits from large-scale cross-robot force corpora is untested. Two additional caveats follow from the experimental design. All results are on a single robot platform with three tasks, 18 trials each, so the reported success rates carry nontrivial sampling uncertainty (a difference of a few trials changes the percentages substantially). The token-count explanation rests on an assumed interaction with $h_t$0's pretrained token budget, and the randomized noise pre-padding is disabled at inference, meaning the encoder operates at deployment on a distribution (no noise prefix) that differs slightly from its training distribution. Open questions include how force memory composes with language-conditioned counting over longer horizons, and whether the frozen-VAE design outperforms joint end-to-end training when substantially more force data are available.

Conclusion

FM-VLA demonstrates that a compact, reconstruction-pretrained latent summary of the wrench history, injected as conditioning tokens into the action expert, enables VLA policies to solve non-Markovian contact-rich tasks that defeat both memoryless policies and visual-memory baselines. The combination of a large empirical margin (83.3% vs. 53.7% for the strongest baseline), negligible inference overhead (+3.3 ms), and ablations isolating the necessity of both force memory and short state context makes a well-supported case for force-based memory as a complement to visual memory in manipulation. The main open issues — fixed latent capacity, single-platform evaluation, and the unverified mechanism behind the token-count degradation — define the boundaries of the current result.