UniFS: Unified Fast-to-Slow Hierarchical Architecture for Vision-Language-Action Models

Abstract: Mainstream Fast-Slow dual system vision-language-action models decouple a high-frequency action expert from a low-frequency vision-LLM for efficiency, yet they face a fundamental frequency dilemma: large update gaps cause semantic drift from stale context, while small gaps erode the intended computational savings. Moreover, because the action expert receives only the VLM's final-layer representation at a single fixed frequency, rich intermediate features are discarded, limiting both information coupling and manipulation precision. Inspired by multi-timescale neural processing in the human brain, we introduce UniFS, a unified fast-to-slow architecture that resolves these challenges through three key designs. First, we stratify the VLM layers into groups with progressively decreasing update frequencies, enabling shallow layers to capture fast-changing dynamics while deeper layers cache stable semantic context. Second, a latent vector inversion mechanism re-routes the interaction order between multi-scale VLM features and the action expert, aligning fast-varying representations with fine-grained action decoding and slow-varying ones with coarse planning. Third, a multi-level supervision strategy enforces a coarse-to-fine learning hierarchy across temporal scales. Together, these designs enable richer cross-frequency information transfer within a single backbone, while the low-frequency pathways additionally preserve temporal context across steps. Experiments on LIBERO show that UniFS achieves state-of-the-art performance (98.3\% average success rate, a 2.5\% gain over VLA-Adapter baseline) while reducing average inference latency from 36.5~ms to 17.8~ms (2.1$\times$ speedup). Real-robot experiments on a Franka platform further validate its practical applicability. Code is opensourced at https://github.com/linsun449/UniFS.

• The paper introduces UniFS, a unified fast-to-slow hierarchy with latent vector inversion and multi-level supervision that replaces rigid fast-slow VLA modules.
• UniFS achieves 98.3% average success on LIBERO and reduces inference latency from 36.5 ms to 17.8 ms, delivering a 2.1× speedup over VLA-Adapter.
• The paper shows that all three components are essential, while cached slow-frequency features can provide implicit short-term memory for real-world robotic manipulation.

Motivation and problem statement

Vision-language-action (VLA) models built on large VLM backbones face a fundamental latency bottleneck: end-to-end control frequencies of 1–10 Hz fall well below the >50 Hz threshold typically required for stable closed-loop manipulation. The prevailing remedy, the Fast-Slow dual-system paradigm, decouples a low-frequency VLM planner from a high-frequency action expert. This paper identifies two structural deficiencies in that paradigm. First, the frequency dilemma: with a single fixed exchange rate between modules, large update gaps cause semantic drift from stale context, while small gaps erode the intended computational savings. Second, information coupling loss: because the action expert receives only the VLM's final-layer representation at one fixed frequency, rich intermediate features are discarded.

The authors ground their design in an empirical observation rather than only biological analogy: measuring cosine distance between layer-wise latent representations across consecutive timesteps in $\pi_0$ and VLA-Adapter reveals that temporal variability differs substantially across depth — deeper layers fluctuate more due to proximity to fast-frequency action supervision. This motivates varying update frequency smoothly across the model instead of tying frequency to specific modules.

Architecture

UniFS comprises three components operating on a single backbone (Qwen2.5-0.5B LLM, DINOv2 + SigLIP visual encoders, and a 24-layer transformer action expert), built on the VLA-Adapter baseline.

Fast-to-Slow Architecture (FSA). The VLM layers are stratified into $K$ groups with progressively decreasing update frequencies $n_1 < n_2 < \cdots < n_K$. Layer group $k$ is recomputed only when $t \equiv 0 \pmod{n_k}$; otherwise its cached output is reused. In practice the vision encoders and LLM are both structured this way, with relative frequencies of $16f$, $8f$, $4f$ for DINOv2/SigLIP and $4f$, $2f$, $K$0 for the LLM, yielding five groups at timescales $K$1.

Latent Vector Inversion (LVI). A key tension arises here: empirical feature dynamics show deeper layers vary fastest, conflicting with FSA's requirement that deeper layers be slow and cacheable. LVI resolves this by inverting the interaction order between multi-scale VLM features and the action expert: initial noisy action proposals cross-attend to deeper (slow) features, while refined outputs interact with shallower (fast) latent vectors. This shifts high-frequency dynamics to shallow layers, leaving deep layers stable and cacheable without sacrificing action responsiveness. Notably, after training with UniFS, the layer-wise dynamics invert as predicted — shallow layers exhibit high temporal variability while deep layers become compact and stable — providing direct evidence that the mechanism works as intended.

Multi-Level Supervision (MLS). Because inversion can create shortcuts through fast-frequency components, auxiliary L1 losses are applied at each frequency group of the action expert via a shared lightweight MLP decoder, averaged uniformly across levels. Low-frequency groups receive identical cached features but different ground-truth actions across timesteps, encouraging coarse long-horizon planning; high-frequency groups learn fine-grained motor corrections.

Training methodology

Two training-time techniques make asynchronous execution practical. A temporal batch sampling strategy draws sorted random timesteps within a contiguous window of each trajectory, preserving causal dependencies within batches while retaining stochasticity across batches. More importantly, Frequency Feature Replacement (FFR) sidesteps the fact that naively skipping layer computation breaks static computation graphs and destroys GPU parallelism: all layer features for all timesteps are computed in a fully parallel forward pass, then retrospectively aligned to prescribed frequencies by copying block-start features to subsequent timesteps ($K$2) via differentiable indexing. This decouples computation parallelism from update asynchrony while preserving gradient fidelity. A consequence worth noting: training gains no speedup from FFR — full forward passes remain necessary — so the acceleration benefit exists only at inference.

Results

On LIBERO (130 tasks across four suites), UniFS achieves a 98.3% average success rate, surpassing the modified VLA-Adapter baseline (95.8%) by 2.5 points and exceeding prior state-of-the-art systems including EO-1 (98.2%), X-VLA (98.1%), and CogVLA (97.4%). Per-suite results are 99.6% (Spatial), 99.6% (Object), 98.1% (Goal), and 95.6% (Long).

Method	Spatial	Object	Goal	Long	Avg.
EO-1	99.7	99.8	99.2	94.8	98.2
X-VLA	98.2	98.6	97.8	97.6	98.1
CogVLA	98.5	98.8	96.5	95.2	97.4
VLA-Adapter†	97.2	98.8	93.4	93.6	95.8
UniFS	99.6	99.6	98.2	95.6	98.3

On efficiency, UniFS attains an average inference latency of 17.8 ms versus 36.5 ms for VLA-Adapter, a 2.1× speedup (mean throughput 449.4 Hz), with per-step latency ranging from 12.3 ms (fastest blocks only) to 32.6 ms (full inference at cycle start). Timing analysis shows each additional slow block contributes roughly 4–5 ms, and the theoretical speedup estimate is about 2.6×.

The ablation results are instructive and somewhat counterintuitive. Enabling FSA alone causes a severe collapse to 70.3% average success — including a drop from 93.6% to 28.4% on LIBERO-Long — which the authors attribute to frequency mismatch between multi-scale latents and the action expert. Adding LVI restores performance to 94.3%, and MLS brings it to 98.3%. Thus the architecture's headline gain depends critically on all three components cohering; FSA in isolation is actively harmful. A further ablation on frequency configurations shows robustness up to a maximum-to-minimum sampling ratio of 24×, beyond which performance degrades with only marginal latency gains; notably, even uniform sampling (no frequency differentiation) incurs no degradation, indicating the learned representations tolerate inference-time reconfiguration.

Real-robot experiments on a Franka Research 3 platform yield 75% (pick), 60% (stack), and 50% (exchange boxes) success rates, comparable to the non-frequency baseline on the first two tasks. The exchange-boxes task is the most meaningful test: it requires short-term memory because correct actions are ambiguous from current observations alone, and UniFS outperforms the baseline there by exploiting cached low-frequency hidden states as an implicit memory mechanism — a capability absent in mainstream dual-system designs that require explicit memory buffers.

Limitations and open questions

The paper concedes several limitations plainly. Training receives no computational speedup, since FFR requires fully parallel forward passes; stabilizing multi-frequency learning demands relatively large batch sizes to ensure sufficient temporal diversity. The architecture and inversion introduce distribution shifts that may interfere with semantic knowledge in pre-trained backbones, requiring careful fine-tuning — an unresolved concern given that the current work uses a small 0.5B backbone rather than larger foundation models. Whether these benefits transfer to stronger backbones such as $K$3, and how multi-frequency adaptation affects pre-trained representations at scale, remains open. Additionally, real-world success rates (50–75%) lag simulation substantially, and the evaluation covers only short-horizon tabletop tasks on a single embodiment.

Conclusion

UniFS replaces rigid module-level frequency separation with smooth, layer-wise frequency stratification inside a single VLM, coupled with inverted feature routing and multi-level supervision. The combination yields state-of-the-art LIBERO performance (98.3%) alongside a 2.1× inference speedup over its own strong baseline, and demonstrates implicit short-term memory through cached slow-frequency pathways. The central open question is whether the approach preserves its advantages when scaled to larger pre-trained VLM backbones, where the distribution-shift effects the authors acknowledge may be more consequential.