Temporally Conditioned Memory-Fusion Policies

• The paper introduces TFP, a framework that injects continuous-time belief updates directly into action decoders to improve stage-aware visuomotor control.
• TFP leverages Liquid Time-Constant dynamics to modulate memory retention based on elapsed time, ensuring adaptive belief updates during critical manipulation events.
• Empirical results show that TFP enhances robustness and reduces stage errors in challenging tasks, outperforming reactive baselines under noisy, occluded conditions.

Searching arXiv for the TFP paper and closely related memory-augmented policy work to ground the article in current literature. Temporally Conditioned Memory-Fusion Policies (TFP) are a memory-action framework for Vision–Language–Action (VLA) policies designed for stage-dependent visuomotor control, especially when visually similar observations correspond to different latent task stages. In TFP, the policy maintains an episode-local task-progress belief with Liquid Time-Constant dynamics and injects the updated belief directly into the flow-matching action decoder through adaptive modulation, so that temporally accumulated context shapes the generated action chunk rather than serving only as passive history context. The method was introduced for manipulation settings in which reactive policies based only on the current observation, instruction, and proprioceptive state break down under occlusion, visual ambiguity, and dependence on prior interaction outcomes (Liang et al., 9 Jul 2026).

1. Problem setting and conceptual scope

TFP addresses a failure mode of reactive VLA policies such as $\pi_{0.5}$ and OpenVLA: the assumption that the next action can be predicted from the most recent observation, instruction, and robot state. The paper argues that this assumption is inadequate in stage-dependent manipulation, where visually similar states may require different actions depending on latent task progress and previous interaction outcomes. Typical examples include settings in which an object may already have been picked up, placed, touched, or released, even though the current image alone does not disambiguate the correct next action (Liang et al., 9 Jul 2026).

The central claim is not merely that manipulation requires memory, but that it requires dynamics-aware belief updates. TFP is built around the idea that a policy should preserve task progress during stable or occluded phases and revise its belief near contact, release, or subgoal transitions. This distinguishes TFP from two families of baselines described in the paper. First, reactive VLA policies either have no memory or rely on attention-based retrieval from recent context, which does not establish a compact internal belief. Second, recurrent baselines such as GRU- or LSTM-based systems maintain a hidden state, but their retention mechanism is tied to policy steps rather than elapsed real time. TFP instead uses elapsed physical time $\Delta t_t$ and channel-wise adaptive time constants, and it supports adaptive policy query intervals at inference, so memory updates can align with robotic execution rather than with a fixed step clock (Liang et al., 9 Jul 2026).

A common misconception is to treat memory in VLA systems as a retrieval problem only. TFP rejects that framing. Its memory is an episode-local belief state intended to encode task progress and previous outcomes in a fixed-size latent vector, and that belief is used to steer action generation directly. This suggests a narrower but more explicit target than generic long-context conditioning: TFP is aimed at stage-aware control under partial observability, not at exhaustive episodic recall.

2. Belief dynamics and temporal conditioning

At each query time $t$, the policy receives the visual observation $\mathcal{I}_t$, robot state $\mathbf{s}_t$, elapsed time since the last query $\Delta t_t$, instruction $\ell$, and previous belief $\mathbf{h}_{t-1}$. Visual tokens are denoted

$$V_t = [\mathbf{v}_{t,1}, \ldots, \mathbf{v}_{t,N}] .$$

The belief update is based on Liquid Time-Constant (LTC) dynamics. A compact representation $\mathbf{x}_t$ combining vision and proprioception is used to compute a candidate belief and a time constant:

$\Delta t_t$0

$\Delta t_t$1

The retention coefficient is then

$\Delta t_t$2

and the belief update becomes

$\Delta t_t$3

The paper defines the write gain as

$\Delta t_t$4

This formulation makes the belief explicitly continuous-time: the degree of retention is a function of elapsed real time and the learned time constants, not merely the number of policy invocations. Each channel has its own adaptive time constant, so some channels can retain slow-varying task-phase information while others can respond more rapidly to transient evidence such as contact or release. The paper interprets large write gain as aggressive belief revision and small write gain as belief preservation, which is especially useful when visual input is ambiguous, noisy, or occluded (Liang et al., 9 Jul 2026).

The significance of this design is empirical as well as architectural. Mechanistic analysis in the paper reports that write-gain changes near manipulation events are about $\Delta t_t$5 larger than in far non-event phases. In the paper’s interpretation, the belief state is therefore event-sensitive rather than uniformly reactive across time. This suggests that TFP uses memory as a selective update process rather than as a dense accumulation of every frame.

3. Memory fusion inside the action decoder

TFP’s “memory-fusion” designation is tied to how the belief enters action generation. The updated belief $\Delta t_t$6 is first projected into decoder space,

$\Delta t_t$7

and then combined with the timestep embedding during flow-matching decoding,

$\Delta t_t$8

At decoder layer $\Delta t_t$9, the paper applies adaptive layer normalization:

$t$0

The stated rationale is that the belief should modulate decoder activations at every layer so that memory directly shapes the generated action chunk. This is the core distinction from approaches in which memory is merely appended as additional tokens or treated as an auxiliary context source. In TFP, temporally accumulated belief is fused into the action pathway itself (Liang et al., 9 Jul 2026).

Training uses contiguous temporal chunks rather than shuffled steps in order to preserve episode-level hidden-state continuity. The paper names this regime Episode-Aware Temporal Batching (EATB): batches parallelize over multiple episodes, each with its own memory state, updating memory only at episode boundaries while preserving cross-chunk state. The objective is the standard flow-matching imitation loss across all sampled chunks. Within the paper’s framing, EATB is necessary because step shuffling would destroy the temporal continuity that the belief state is meant to encode.

This architecture establishes a specific interpretation of “temporally conditioned memory fusion.” The memory is conditioned on elapsed time and current sensory context; the fusion is performed through adaptive decoder modulation; and the output affected by that fusion is not a retrieved summary but the action chunk itself.

4. Empirical performance

TFP is evaluated on LIBERO, LIBERO-plus, and the memory-focused MIKASA ShellGameTouch diagnostic, with additional real-robot experiments on the Galaxea A1 platform. With a 3.3B-parameter model, the paper reports that TFP improves the average success rate from $t$1 to $t$2 on LIBERO and from $t$3 to $t$4 on LIBERO-plus. On the LIBERO Long split, TFP reaches $t$5 compared with $t$6 for $t$7. On LIBERO-plus, the paper notes the biggest improvements in noise and lighting perturbations. On MIKASA ShellGameTouch, TFP achieves success up to $t$8 (Liang et al., 9 Jul 2026).

Setting	Comparison	Result
LIBERO average	$t$9 $\mathcal{I}_t$0 vs TFP $\mathcal{I}_t$1	TFP higher
LIBERO-plus average	$\mathcal{I}_t$2 $\mathcal{I}_t$3 vs TFP $\mathcal{I}_t$4	TFP higher
LIBERO Long split	$\mathcal{I}_t$5 $\mathcal{I}_t$6 vs TFP $\mathcal{I}_t$7	TFP higher
MIKASA ShellGameTouch	OpenVLA-OFT $\mathcal{I}_t$8 vs TFP $\mathcal{I}_t$9	TFP higher
MIKASA ShellGameTouch	MemoryVLA $\mathbf{s}_t$0 vs TFP $\mathbf{s}_t$1	MemoryVLA higher

The paper also reports real-robot gains. On object swap, TFP achieves $\mathbf{s}_t$2 successes versus $\mathbf{s}_t$3 for the baseline, with the reported improvement attributed to major reduction in failures due to wrong stage or progress recall. On counting pick-place, TFP reaches $\mathbf{s}_t$4 versus $\mathbf{s}_t$5 for the baseline. All results are reported with Wilson $\mathbf{s}_t$6 confidence intervals, and the paper states that the gains are statistically significant, especially on long-horizon and robustness splits (Liang et al., 9 Jul 2026).

These results position TFP as a method whose strongest empirical gains arise in settings where stage ambiguity and perturbation robustness matter. A plausible implication is that the architecture is particularly useful when errors arise not from low-level actuation alone but from confusion about latent task phase.

5. Mechanistic evidence, interpretation, and limitations

The paper supplements benchmark results with mechanistic analysis intended to show that the belief state is causally involved in behavior rather than functioning as an inert auxiliary vector. Two findings are central. First, write-gain changes near manipulation events are about $\mathbf{s}_t$7 larger than during far non-event phases. Second, hidden-state interventions that fix the observation while varying the hidden state lead to substantially different predicted action chunks. The paper therefore concludes that the belief causally modulates generated action chunks (Liang et al., 9 Jul 2026).

The same section supports the claim that TFP stabilizes task progress during ambiguous phases. When observational evidence is weak, stable memory channels preserve belief and avoid spurious overwriting; when decisive evidence arrives near subgoal transitions, write gain increases and the belief can shift rapidly. In the paper’s interpretation, this yields robustness to occlusion and noise while reducing stage errors such as unnecessary repetitions or confusion about whether a subtask has already been completed.

The paper also states a limitation. TFP’s memory is optimized for continuous event-belief tracking rather than for object-centric symbolic bindings. This is used to explain why TFP, despite outperforming OpenVLA-OFT on MIKASA ShellGameTouch, is outperformed by MemoryVLA on that diagnostic ($\mathbf{s}_t$8 versus $\mathbf{s}_t$9). The cited example is a task requiring knowledge of which cup hides the ball: such behavior may require a more object-structured memory than the fixed-size belief used by TFP. This is an important boundary condition, because it distinguishes latent task-progress tracking from explicit identity-location assignment.

A second misconception is therefore that any temporally conditioned memory mechanism should dominate all memory benchmarks. The reported results do not support that. TFP is strongest where continuous task progress and event-sensitive updating dominate the control problem; it is not presented as a universal replacement for object-structured or retrieval-heavy memory systems.

6. Position within temporally conditioned memory and fusion research

TFP belongs to a broader class of systems that make temporal structure explicit in memory selection, consolidation, or policy conditioning, but it occupies a distinctive design point. In robot manipulation, KEMO proposes an event-driven keyframe memory for VLA policies that automatically selectively preserves keyframes associated with task-relevant state changes, encodes them as compact temporally ordered memory tokens, and integrates them with current visual features through cross-attention and gated residual fusion. KEMO improves aggregate Task Success Rate by $\Delta t_t$0 and Stage Completion Rate by $\Delta t_t$1 over a memory-free baseline, and its ablations show that event-driven keyframe selection outperforms uniform sampling and recent-frame retention (Zeng et al., 22 Jun 2026). Mem-World, by contrast, uses a 4D wrist-view-centered surfel-indexed memory for action-conditioned world models, retrieving history frames according to future actions and predicted future wrist viewpoint; it improves the Pearson correlation with real-world performance by $\Delta t_t$2 and increases long-horizon task success from $\Delta t_t$3 to $\Delta t_t$4 through synthetic data generation (Zheng et al., 17 Jun 2026). Relative to these systems, TFP does not store an explicit bank of keyframes or retrieved history frames for inference-time fusion; it maintains a compact episode-local belief and injects it into the decoder.

In reinforcement learning, an earlier precursor is Diverse Trajectory-Conditioned Self-Imitation Learning, which stores diverse trajectories in a buffer and conditions a policy on a full demonstration sequence rather than on goals alone. That method is designed for sparse-reward exploration and reports state-of-the-art scores under five billion frames on Montezuma’s Revenge and Pitfall without expert demonstrations or resetting to arbitrary states (Guo et al., 2019). The connection is structural rather than identical: both methods reject purely reactive control and treat temporally extended context as policy input, but TFP replaces explicit trajectory conditioning with a latent continuous-time belief.

Outside robotics, several recent memory systems make time a first-class organizing principle. TiMem organizes conversational memory through a Temporal Memory Tree with five levels from segment to profile, using semantic-guided consolidation and complexity-aware recall; it reaches $\Delta t_t$5 on LoCoMo and $\Delta t_t$6 on LongMemEval-S while reducing recalled memory length by $\Delta t_t$7 on LoCoMo (Li et al., 6 Jan 2026). Temporal Semantic Memory constructs a semantic timeline and durative memories for personalized LLM agents and reports up to $\Delta t_t$8 absolute improvement in accuracy on LongMemEval and LoCoMo (Su et al., 12 Jan 2026). Memory-T1 learns a time-aware memory selection policy using reinforcement learning for multi-session dialogue, guided by answer accuracy, evidence grounding, and temporal consistency rewards, and achieves $\Delta t_t$9 on Time-Dialog while remaining robust up to $\ell$0k tokens (Du et al., 23 Dec 2025). In time-series forecasting, TS-Memory distills an offline retrieval teacher into a lightweight adapter and fuses memory and backbone predictions with constant-time overhead, avoiding inference-time datastore search (Lyu et al., 12 Feb 2026).

Taken together, these systems indicate that “temporally conditioned memory-fusion policy” is not a single algorithmic template but a family of designs in which temporal structure governs what is retained, when memory is revised, and how memory influences downstream inference. TFP’s particular contribution within that family is to couple continuous-time belief dynamics with direct decoder modulation for visuomotor action generation, thereby treating latent task progress as an internal control variable rather than as a retrieved external context (Liang et al., 9 Jul 2026).