PRIMT: Multimodal Preference RL

• PRIMT is a preference-based RL framework that replaces extensive human labeling by using hierarchical neuro-symbolic fusion for synthetic multimodal feedback.
• It combines LLM and VLM outputs via intra- and inter-modal fusion to generate reliable preference labels and improve credit assignment through bidirectional trajectory synthesis.
• Empirical findings show that PRIMT converges 2–3× faster and achieves 10–20 pp higher success rates on robotics tasks compared to FM-only baselines.

Searching arXiv for PRIMT and closely related preference-based RL work to ground the article. arXiv_search(query="PRIMT Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation Models", max_results=5, sort_by="relevance") arXiv_search(query="preference-based reinforcement learning multimodal feedback trajectory synthesis foundation models robotics", max_results=10, sort_by="relevance") PRIMT—“Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation Models”—is a preference-based reinforcement learning (PbRL) framework for teaching robots complex behaviors without reward engineering. It is designed to address three stated difficulties in standard PbRL: extensive human input, query ambiguity, and credit assignment. Its central design combines a hierarchical neuro-symbolic fusion strategy for multimodal synthetic feedback with bidirectional trajectory synthesis, namely foresight trajectory generation and hindsight trajectory augmentation, while learning a parametric reward model and updating the policy via SAC (Wang et al., 19 Sep 2025).

1. Problem formulation and motivation

PRIMT is formulated in a standard Markov decision process with state space $\mathcal S$, action space $\mathcal A$, transition kernel $P(s_{t+1}\mid s_t,a_t)$, and unknown sparse reward. In the PbRL setting, the reward $r(s,a)$ is not hand-crafted. Instead, the system collects comparative feedback on pairs of trajectories. A trajectory of length $T$ is

$\sigma=\{(s_1,a_1),\dots,(s_T,a_T)\}.$

Given two trajectories $\sigma^A,\sigma^B$, an annotator returns a preference label

$\Upsilon\in\{-1,0,1\},$

where $\Upsilon=1$ denotes $\sigma^A$ preferred over $\mathcal A$0, $\mathcal A$1 denotes $\mathcal A$2 preferred over $\mathcal A$3, and $\mathcal A$4 denotes indecision.

The reward model $\mathcal A$5 is fitted by minimizing the cross-entropy between predicted and synthetic preference labels under the Bradley–Terry model:

$\mathcal A$6

The motivation for PRIMT follows directly from three limitations identified for standard PbRL. First, real humans must label thousands of pairs. Second, early trajectories are noisy or random, so most comparisons yield indecision or otherwise uninformative labels. Third, trajectory-level labels do not specify which states or actions caused the preference. PRIMT is explicitly organized around these three failure modes: replacing direct human feedback with synthetic foundation-model feedback, reducing early-stage query ambiguity through warm-started trajectory generation, and improving credit assignment through counterfactual trajectory augmentation (Wang et al., 19 Sep 2025).

2. System architecture and learning loop

PRIMT consists of two synergistic modules: hierarchical neuro-symbolic fusion of multimodal synthetic feedback, and bidirectional trajectory synthesis through foresight and hindsight. The high-level loop begins by initializing the replay buffer with trajectories generated by LLM code generation, then appending random trajectories from the initial policy. Candidate pairs are sampled from the buffer by uncertainty. For each pair, multimodal fusion returns a label and confidence; clear labels are added to the preference dataset, and high-confidence labels can trigger hindsight augmentation through counterfactual pair generation. The reward model is then updated by minimizing a preference loss plus a causal auxiliary loss, the replay buffer is relabeled with the learned reward, and the policy is updated via SAC.

This organization makes PRIMT a PbRL pipeline in which foundation models are used in three different roles rather than only one. They are used as annotators for synthetic preference labels, as generators of warm-start trajectories, and as generators of counterfactual edits. A plausible implication is that PRIMT treats foundation models not merely as evaluators of behavior but as components that modify the data distribution seen by the reward learner.

3. Hierarchical neuro-symbolic fusion for multimodal feedback

The multimodal feedback mechanism is hierarchical because it performs both intra-modal and inter-modal fusion. The intra-modal stage queries an LLM using a textual projection $\mathcal A$7 of state–action sequences in a chain-of-thought prompt, repeating the query $\mathcal A$8 times with random order. From these repetitions, PRIMT computes a majority-vote label

$\mathcal A$9

Confidence aggregation combines average confidence on agreeing votes and vote consistency:

$P(s_{t+1}\mid s_t,a_t)$0

$P(s_{t+1}\mid s_t,a_t)$1

The VLM branch operates analogously, but on a small set of keyframes $P(s_{t+1}\mid s_t,a_t)$2 extracted via zero-velocity, residual-peak, and change-point methods, again using repeated chain-of-thought queries. This gives PRIMT two separate synthetic judgments, one grounded in textualized state–action trajectories and the other grounded in visual summaries.

The inter-modal stage represents a set of atoms,

$P(s_{t+1}\mid s_t,a_t)$3

with

$P(s_{t+1}\mid s_t,a_t)$4

These atoms are fused by Probabilistic Soft Logic (PSL) with weighted rules. The stated rules include an agreement rule,

$P(s_{t+1}\mid s_t,a_t)$5

and conflict-resolution rules that prioritize the VLM if VDHigh is large, the LLM if TDHigh is large, and otherwise indecision. Inference solves a convex optimization to yield soft scores $P(s_{t+1}\mid s_t,a_t)$6, and the final label is chosen by $P(s_{t+1}\mid s_t,a_t)$7.

The significance of this design lies in the fact that PRIMT does not rely on a single-modality FM evaluation. The stated claim is that combining LLM and VLM feedback with neuro-symbolic fusion yields more reliable synthetic preferences than single-modality approaches. The ablation results further state that inter-modal fusion through PSL is critical for high label accuracy (Wang et al., 19 Sep 2025).

4. Bidirectional trajectory synthesis: foresight and hindsight

The foresight component is intended to reduce early-stage query ambiguity. Its goal is to bootstrap the replay buffer with diverse, task-aligned trajectories from an LLM. The specified procedure is: prompt the LLM to produce a high-level plan; translate each step into executable Python code snippets as motion primitives; execute the code under varied initial conditions to obtain $P(s_{t+1}\mid s_t,a_t)$8 trajectories; insert these into the replay buffer as demonstrations; and then use uncertainty-based sampling to pair them with random or exploration trajectories so that comparisons become informative.

The hindsight component addresses credit assignment through counterfactual reasoning. Whenever a pair $P(s_{t+1}\mid s_t,a_t)$9 receives a clear label with high confidence, where $r(s,a)$0, the LLM is asked through an SCM-inspired prompt to identify critical causal steps $r(s,a)$1 in $r(s,a)$2, apply minimal edits to one critical step to produce a counterfactual $r(s,a)$3 that reverses the preference, and verify $r(s,a)$4 via the LLM module. Only valid counterfactuals are accepted, and the pair $r(s,a)$5 is stored in the counterfactual dataset.

Taken together, foresight and hindsight form the “bidirectional trajectory synthesis” portion of PRIMT. The forward-looking stage changes the initial trajectory distribution before preference collection, while the backward-looking stage edits successful trajectories after a clear preference has been established. This suggests a deliberate decomposition: foresight targets query ambiguity in the data collection phase, whereas hindsight targets credit assignment in reward learning (Wang et al., 19 Sep 2025).

5. Reward learning objective and causal auxiliary loss

PRIMT augments standard preference learning with a causal auxiliary loss defined on counterfactual pairs. Let $r(s,a)$6 if step $r(s,a)$7 was edited and $r(s,a)$8 otherwise. The auxiliary loss is

$r(s,a)$9

The total reward-learning loss is

$T$0

where $T$1 is the cross-entropy over trajectory pairs in $T$2, and $T$3 balances scales.

The structure of this loss is explicit. Edited steps are trained with a causal contrast term so that the reward model reflects the change that reversed preference. Unedited steps are trained with a reward-consistency term so that unaffected portions of the trajectory remain aligned across the original and counterfactual variants. Within the stated framing, this is the mechanism by which hindsight augmentation improves credit assignment.

The reported reward-alignment result is that the learned $T$4 from PRIMT tracks ground-truth reward closely at the state-action level, whereas baselines produce noisy or uncalibrated signals. The accompanying ablations state that hindsight together with the causal auxiliary loss is important for peak performance (Wang et al., 19 Sep 2025).

6. Experimental scope, empirical findings, and interpretive limits

PRIMT is evaluated on 2 locomotion and 6 manipulation tasks. The locomotion tasks on DeepMind Control are Hopper Stand and Walker Walk. The manipulation tasks on MetaWorld are Button Press, Door Open, and Sweep Into. The manipulation tasks on ManiSkill2 are PickSingleYCB, StackCube, and PegInsertionSide.

The reported baselines are RL-VLM-F, described as vision-only; RL-SaLLM-F, described as LLM-only with self-augmentation; PrefCLM, described as crowdsourced LLM ensembles; PrefMul, described as naïve multimodal feed to one FM; and PrefGT, described as oracle from ground-truth reward. The ablations are w/o Intra-fusion, w/o Inter-fusion, w/o Foresight, w/o Hindsight, and w/o Causal-aux.

Category	Items
Locomotion (DeepMind Control)	Hopper Stand; Walker Walk
Manipulation (MetaWorld)	Button Press; Door Open; Sweep Into
Manipulation (ManiSkill2)	PickSingleYCB; StackCube; PegInsertionSide

The key quantitative findings are stated as follows: PRIMT converges up to 2–3× faster and achieves 10–20 pp higher success rates than any FM-only baseline; it matches or exceeds PrefGT on several tasks, including PegInsertionSide, despite having no oracle reward; it reduces indecision rates by 40–60%; and it increases synthetic-label accuracy, measured against PrefGT, to 85–90%. The real-robot result is that PRIMT successfully deploys block lifting and stacking policies on a Kinova Jaco arm, outperforming PrefCLM in both success rate and safety (Wang et al., 19 Sep 2025).

The limitations are also explicit. PRIMT depends on FM API calls, with associated compute cost and latency. Its current instantiation is single-agent robotics, and generalizing to multi-agent or open-world domains remains open. Quality bounds hinge on FM capacity; an ablation on “gpt-4o-mini” shows graceful degradation. Future work identified for the framework includes extending multimodal fusion to audio, haptics, or text demonstrations, end-to-end fine-tuning of foundation models within the PbRL loop, and applying PRIMT principles to sequential decision tasks beyond robotics, including text planning and human–AI interaction.

A common misconception would be to read PRIMT as removing supervision altogether. The stronger factual statement is narrower: the framework nearly closes the gap to an oracle teacher without any human labels, but its performance depends on the reliability, latency, and capacity of foundation models. Another possible misconception is to treat the multimodal component as a simple concatenation of modalities; the formal design instead uses hierarchical intra-modal aggregation and inter-modal PSL-based fusion. These distinctions are central to the framework’s stated contribution and to the interpretation of its empirical results.