TFP: Temporally Conditioned Memory-Fusion Policies for Visuomotor Learning

Abstract: Vision--Language--Action (VLA) policies such as $π_{0.5}$ and OpenVLA perform well on many manipulation tasks, but they are often reactive: the next action is predicted from the current observation, instruction, and proprioceptive state. This assumption breaks down in stage-dependent manipulation, where visually similar states may require different actions depending on latent task progress and previous interaction outcomes. We argue that such tasks require not only memory, but dynamics-aware belief updates: the policy should preserve task progress during stable or occluded phases and revise its belief near contact, release, or subgoal transitions. We introduce Temporally Conditioned Memory-Fusion Policies (TFP), a lightweight memory-action framework for VLA backbones. TFP maintains an episode-local task-progress belief with Liquid Time-Constant dynamics and injects the updated belief directly into the flow-matching action decoder through adaptive modulation. This lets temporally accumulated context shape the generated action chunk, rather than serving only as passive history context. With a 3.3B-parameter model, TFP improves the average success rate from (96.9\%) to (98.75\%) on LIBERO and from (91.4\%) to (93.77\%) on LIBERO-plus. On the memory-focused MIKASA ShellGameTouch diagnostic, TFP achieves success up to (75.0\%). Mechanistic analyses show that write-gain changes near manipulation events are about (6\times) larger than far non-event phases, and hidden-state interventions show that the belief causally modulates generated action chunks. These results suggest that compact, event-sensitive memory dynamics can improve VLA policies under occlusion, visual perturbation, and stage-dependent task structure.

• The paper introduces a novel TFP architecture that uses an LTC-based continuous memory to track task progress for improved visuomotor control.
• It employs Episode-Aware Temporal Batching to maintain hidden state continuity, enabling effective training over long sequences.
• TFP achieves higher success rates on benchmarks, particularly enhancing performance in long-horizon, occlusion-prone manipulation tasks.

Temporally Conditioned Memory-Fusion Policies for Visuomotor Learning

Background and Motivation

Visuomotor robot policies that map observations and language instructions directly to actions, such as $\pi_{0.5}$, OpenVLA, and Octo, have achieved notable capability in multitask and generalist settings. However, these Vision–Language–Action (VLA) policies typically operate in a reactive regime, producing each action solely from the current observation, proprioceptive state, and instruction. This reactivity becomes inadequate for stage-dependent, memory-intensive manipulation tasks—such as object swapping or long-horizon procedures—where different actions might be required in visually similar scenes depending on the latent stage of the task or previous agent interactions.

The underlying challenge is that correct action selection often depends on an internal, temporally coherent belief over task progress and recent interaction outcomes. This belief must be robust to periods of ambiguous or occluded observations, maintaining consistency through stability, and should rapidly update when pivotal manipulation events (e.g., contact, release, subgoal transitions) are detected.

Figure 1: Overview illustrating the motivation for memory-conditioned visuomotor control.

Methodological Advances

TFP Architecture

The paper introduces Temporally Conditioned Memory-Fusion Policies (TFP), a memory-augmented framework for VLA robotic policies. TFP maintains an explicit, episode-local latent task-progress belief, parameterized as a continuous-time latent state updated via Liquid Time-Constant (LTC) network dynamics. This update is driven by visual and proprioceptive observations and modulated by the actual elapsed physical time between policy queries. Crucially, this belief is not merely auxiliary context—it directly modulates the action-decoder layers through adaptive normalization (AdaLN), conditioning action chunk generation on this dynamic memory.

Figure 2: TFP maintains a continuous-time latent belief and injects it into the action decoder through adaptive modulation.

At policy query $t$, the updated memory $h_t$ results from exponentially relaxing toward a candidate value $\hat{h}_t$ suggested by the current observation and prior belief, with per-channel write gains governed by input-dependent, vector-valued time constants—a direct instantiation of elapsed-time-dependent memory filtering.

Training: Episode-Aware Temporal Batching

Standard random chunked training disrupts episode-local hidden-state continuity, which is necessary for meaningful recurrent memory. TFP addresses this with Episode-Aware Temporal Batching (EATB): multiple episode segments are unrolled in parallel while preserving distinct hidden states, with numeric continuity across chunks and explicit hidden state resets only at episode boundaries. This enables practical recurrent training without the prohibitive compute cost of full-episode backpropagation.

Figure 3: Episode-Aware Temporal Batching trains on contiguous chunks while carrying episode-local hidden states across truncated segments.

Adaptive Inference

In deployment, TFP policies operate with adaptive receding-horizon executors: at each query, the model predicts a full action chunk, but the actual executed prefix before the next policy query can vary depending on risk heuristics (e.g., anticipated transition, instability, or contact). The LTC memory update is passed the real elapsed time, enabling temporally calibrated state evolution.

Empirical Results and Mechanistic Analysis

The principal experimental findings—at parity or smaller scale compared to related models—are:

• On LIBERO: TFP improves average success rate from $96.9\%$ to $98.75\%$, with a notable gain on the challenging long-horizon split ($97.0\%$ vs. $92.4\%$).
• On LIBERO-plus: Robustness to noise and visual perturbation on the extended LIBERO-plus benchmark is improved from $91.4\%$ to $93.77\%$ average success, again most notably under noise and lighting shifts.
• On MIKASA ShellGameTouch: TFP attains $t$0 success, demonstrating substantial memory-dependent reward in highly occluded tasks, though state-of-the-art object-centric memory policies reach higher performance.
• On real-world Galaxea A1 robot: Success rate on memory-dependent object swap improves from $t$1 with $t$2 to $t$3 with TFP, and stage-level memory errors dominate failure cases for the baseline but are largely mitigated by TFP.

Mechanistic Insights and Ablations

TFP's LTC memory is shown to be event-sensitive: analysis reveals that write-gain transitions (i.e., rapid changes in belief update rates) cluster around manipulation events such as grasp, release, and subgoal transitions. These gain changes are $t$4 larger during event windows than non-event phases, supporting the interpretation that memory update is dynamically responsive to policy-relevant state transitions.

Figure 4: LTC belief update gain $t$5 during a rollout, with pronounced gain changes around manipulation events.

Causal interventions demonstrate that holding the observation and instruction fixed but varying the hidden state produces significant variability in the resulting action chunks, directly tying task-progress belief to generated motor behavior.

Figure 5: Action-chunk variation caused by changing only the LTC hidden state under the same observation.

Extensive ablation establishes that:

• TFP's empirical gains are not explained simply by recurrent state or smooth interpolation—continuous-time memory update, elapsed-time calibration, and direct action-head conditioning are all critical.
• Baselines with analogous GRU or SSM (S4D) recurrence perform worse, especially on long-horizon tasks, and removing elapsed-time input from the LTC update sharply reduces performance in scenarios with irregular query intervals.

Theoretical and Practical Implications

Theoretical Implications:

TFP formalizes an action-conditioning memory mechanism wherein (i) latent task-progress is maintained by a continuous-time evolution (exponential relaxation toward perceptually inferred candidate beliefs, with channel- and state-dependent time constants), and (ii) retention and update are directly aligned with event occurrence and actual elapsed time, yielding strong time-consistency and matching task-phase semantics. Notably, exponential retention is shown to be the unique continuous rule consistent with composition over elapsed time intervals, undergirding the memory update's formal coherence.

Practical Implications:

Practically, TFP demonstrates that VLA policies for robotics should not treat memory as merely a context-retrieval mechanism but must instead maintain a dynamic, action-relevant belief that survives ambiguous, delayed, or occluded observations. Injecting temporally-structured latent memory directly into the action-generation process, as opposed to cross-attention over tokenized history, yields superior performance in memory-critical settings without sacrificing overall manipulation skill. Event-aligned write-gain dynamics suggest downstream strategies for automated event boundary detection and actionable subgoal inference.

Limitations and Directions for Future Research

TFP's recurrent fine-tuning still carries significant computational cost due to the need to preserve hidden-state continuity across long horizons, despite the efficiency gains from EATB. Additionally, while the event-sensitive LTC memory improves many general and occlusion-heavy tasks, categorical object-location bindings (as needed for ShellGameTouch) remain relatively challenging, pointing to the need for hybrid strategies integrating object-centric memory representations with continuous-time belief filters.

Scaling to complex scenes (mobile manipulators, compositional multi-object tasks, or dexterous hands), efficient memory fine-tuning, and object-centric memory fusion emerge as natural future directions. Dynamically adjustable temporal abstraction, integration with selective spatial reasoning (e.g., slot attention), and hierarchical memory architectures could further extend TFP's utility.

Conclusion

TFP provides an effective approach to incorporating temporally sensitive, event-aligned memory into VLA visuomotor policies, with minimal computational overhead and structurally grounded inductive bias toward manipulation-stage belief tracking. By fusing continuous-time LTC memory with adaptive, direct action-head conditioning, TFP achieves substantial empirical improvements on both simulated and real-world benchmarks, with diagnostic analyses confirming the functional role and selectivity of the memory module. These results strongly support a design paradigm for robotics policies in which memory is both temporally structured and generatively causal for action planning, laying essential groundwork for further advances in memory-efficient, robust, and generalist robot learning.

---

Reference:

"TFP: Temporally Conditioned Memory-Fusion Policies for Visuomotor Learning" (2607.08283)