OpTI-BFM: Optimistic Task Inference in Zero-Shot RL

• The paper introduces OpTI-BFM, an online procedure that infers unseen reward functions via an optimistic exploration rule over task embeddings.
• It leverages a linear structure between successor features and reward regressions to achieve sublinear regret guarantees in zero-shot reinforcement learning.
• Empirical results show that OpTI-BFM reaches Oracle-level performance rapidly with minimal data overhead on standard DeepMind Control Suite benchmarks.

Searching arXiv for papers explicitly using the term “OpTI-BFM” and closely related “Behavior Foundation Model” work. OpTI-BFM denotes Optimistic Task Inference for Behavior Foundation Models, an online procedure for zero-shot reinforcement learning in settings where the reward function is not directly known, is expensive to label, or cannot be evaluated on a large offline dataset. It is formulated for Behavior Foundation Models (BFMs) built on Universal Successor Features (USFs) / Successor Features (SFs), and replaces the standard test-time regression over a labeled inference dataset with an optimistic exploration rule over task embeddings. The method models uncertainty over the latent reward parameter, selects behaviors according to an upper-confidence criterion, and is accompanied by a regret guarantee connected directly to upper-confidence algorithms for linear bandits (Rupf et al., 23 Oct 2025).

1. Formal object and problem regime

OpTI-BFM is defined in a setting where rewards are assumed linear in a feature map,

$r(s)=z^\top \phi(s),$

for some latent reward parameter $z\in\mathbb{R}^d$. For a policy $\pi$, successor features are

$$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$

so that the action-value function is linear in the same reward embedding,

$Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$

A USF-based BFM then learns a family of parameterized policies $(\pi_z)_{z\in\mathcal Z}$ such that, for each $z\in\mathcal Z$,

$$\pi_z(a\mid s)\in \argmax_a \psi^{\pi_z}(s,a)^\top z.$$

Once the reward embedding $z_r$ is known, a near-optimal policy can therefore be retrieved immediately by conditioning on $z_r$ (Rupf et al., 23 Oct 2025).

The standard BFM inference pipeline performs task inference offline at test time: given a dataset $z\in\mathbb{R}^d$0 with reward labels $z\in\mathbb{R}^d$1, one solves

$z\in\mathbb{R}^d$2

This is computationally cheap, but it presumes access to enough labeled data. The method is motivated by cases in which the pretraining dataset is unavailable or proprietary, reward evaluation is expensive, rewards require costly human labeling, or observations are pixels and reward labeling is nontrivial. In that sense, OpTI-BFM addresses data cost, not merely test-time compute cost.

2. Relation to Behavior Foundation Models

The underlying BFM concept is broader than OpTI-BFM. In the humanoid-control formulation named Behavior Foundation Model, the central claim is that diverse whole-body control settings—locomotion, motion tracking, teleoperation, language, and related regimes—can be reframed as instances of behavior generation conditioned on goal states rather than as isolated task-specific policies. The paper describes BFM as a generative model pretrained on large-scale behavioral datasets and implemented using a masked online distillation framework together with a Conditional Variational Autoencoder (CVAE) to model behavioral distributions (Zeng et al., 17 Sep 2025).

In that formulation, one begins from a goal-conditioned RL view with state

$z\in\mathbb{R}^d$3

reward

$z\in\mathbb{R}^d$4

and PPO objective

$z\in\mathbb{R}^d$5

For pretraining, the perspective shifts from reward maximization to likelihood modeling over behaviors. With dataset

$z\in\mathbb{R}^d$6

the objective becomes

$z\in\mathbb{R}^d$7

and then, after introducing goal states and marginalizing over them, a tractable lower bound based on conditional likelihood is used. This establishes BFM as a reusable behavior prior rather than a single-task controller.

This relationship is conceptually important. OpTI-BFM does not redefine the BFM substrate; it assumes a pretrained BFM already exists and tackles the separate problem of inferring the current task embedding through online interaction. The data block is explicit that the 2025 humanoid paper does not use the term “OpTI-BFM” and calls the model simply BFM. A plausible implication is that OpTI-BFM should be read as an inference layer for BFMs rather than as a synonym for the original behavior-generation architecture.

3. Online task inference as active exploration

OpTI-BFM assumes access to a pretrained BFM providing policies $z\in\mathbb{R}^d$8, successor feature estimates $z\in\mathbb{R}^d$9, and a feature map $\pi$0. Interaction occurs in episodes of horizon $\pi$1. At each step $\pi$2, the agent chooses a task embedding $\pi$3, executes

$\pi$4

and observes next state $\pi$5 and reward $\pi$6. The discounted return of episode $\pi$7 is

$\pi$8

and the regret over $\pi$9 episodes is

$$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$0

where $$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$1 is the return of the optimal task embedding $$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$2 (Rupf et al., 23 Oct 2025).

The key modeling step is to exploit two linear structures simultaneously:

$$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$3

and

$$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$4

Task inference is therefore converted into an online linear optimization problem over the hidden reward parameter $$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$5, where each candidate behavior $$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$6 induces a payoff through successor features. The method maintains a regularized least-squares estimate

$$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$7

or equivalently

$$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$8

From this, OpTI-BFM constructs a confidence ellipsoid

$$\psi^\pi(s_0,a_0)=\sum_{t\ge 0}\gamma^t\,\mathbb{E}\!\left[\phi(s_t)\mid s_0,a_0,\pi\right],$$9

intended to contain the true task embedding $Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$0 with high probability. The inference problem is thus no longer passive regression on a fixed dataset; it becomes an active exploration problem over task embeddings.

4. Optimism principle and bandit-theoretic structure

The defining decision rule of OpTI-BFM is

$Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$1

Here $Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$2 is the task embedding used to condition the BFM policy, while $Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$3 ranges over plausible reward embeddings in the current confidence set. The chosen behavior is therefore the one that could be best under the most favorable reward still consistent with observed data. This is the classical optimism in the face of uncertainty principle in a BFM setting (Rupf et al., 23 Oct 2025).

The paper rewrites the inner maximization in the familiar UCB form: $Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$4 This exposes the method as a direct analogue of linear-UCB selection, with the usual decomposition into estimated value and uncertainty bonus. The algorithmic loop is correspondingly simple: initialize $Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$5; at each step compute an optimistic embedding $Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$6, execute $Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$7, observe $Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$8 and $Q_r^\pi(s_0,a_0)=z^\top \psi^\pi(s_0,a_0).$9, and update $(\pi_z)_{z\in\mathcal Z}$0 from $(\pi_z)_{z\in\mathcal Z}$1.

The paper emphasizes a structural twist relative to standard linear bandits. There are two distinct feature-like objects:

• $(\pi_z)_{z\in\mathcal Z}$2, used for regression on the hidden reward parameter $(\pi_z)_{z\in\mathcal Z}$3;
• $(\pi_z)_{z\in\mathcal Z}$4, used for action selection through the optimism objective.

This separation is technically significant. Reward-level regression exploits per-step feedback, whereas action selection operates on successor features aggregated under the policy induced by the chosen task embedding. The paper explicitly notes that using $(\pi_z)_{z\in\mathcal Z}$5 for regression yields tighter estimates than using episode-level return data alone.

Under assumptions labeled (A1) Perfect USF, (A2) Linear reward, (A3) Optimization oracle, and (A4) Bounded norms, the main theoretical statement is:

$(\pi_z)_{z\in\mathcal Z}$6

The proof uses a confidence radius

$(\pi_z)_{z\in\mathcal Z}$7

together with optimism, a Cauchy–Schwarz step in the $(\pi_z)_{z\in\mathcal Z}$8-norm, an elliptical potential argument, and determinant bounds. The paper further compares the covariance matrix built from reward-level features with the one built from episode-level successor features. With

$(\pi_z)_{z\in\mathcal Z}$9

it proves

$z\in\mathcal Z$0

and therefore

$z\in\mathcal Z$1

This is the bandit-theoretic mechanism behind the sublinear regret guarantee.

5. Empirical characteristics and data efficiency

The empirical study evaluates OpTI-BFM on standard zero-shot RL benchmarks from the DeepMind Control Suite: Walker, Cheetah, and Quadruped, each with four tasks or reward functions. The base BFM is the Forward-Backward (FB) framework, described as a successor-features-based BFM. Reported baselines are OpTI-BFM, OpTI-BFM-TS (a Thompson-sampling variant), LoLA, Random, and Oracle, where Oracle uses labeled data or the true reward to compute the best $z\in\mathcal Z$2 (Rupf et al., 23 Oct 2025).

The reported metrics are episodic return $z\in\mathcal Z$3, relative performance relative to Oracle, and—in a data-efficiency study—the performance of the policy retrieved from the inferred embedding $z\in\mathcal Z$4. The main experimental claims are precise:

• OpTI-BFM reaches Oracle-level performance in about 5 episodes.
• In the reported setting this corresponds to roughly 5k environment steps.
• It outperforms LoLA in early data efficiency.
• The Thompson-sampling variant is competitive but weaker than the optimism-based rule on some tasks, especially Cheetah.
• The UCB version is about 5x slower than just running the policy, while the TS variant is about 4x slower.
• Reported per-step times on an RTX 4090 are 0.772 ms for Oracle, 3.567 ms for OpTI-BFM, and 2.756 ms for OpTI-BFM-TS.

These results support a specific interpretation: the method is not presented as compute-free, but as adding minimal compute overhead relative to the gain in data efficiency. The paper also studies explicit label querying through the information-gain statistic

$z\in\mathcal Z$5

and only requests labels when this quantity exceeds a threshold $z\in\mathcal Z$6. The reported outcome is that labeled data can be reduced by more than an order of magnitude in easier tasks while maintaining performance.

A common misconception would be to treat OpTI-BFM as a black-box fast-adaptation heuristic. The experimental comparison suggests the opposite emphasis: the gain comes from exploiting the linear structure between rewards and task embeddings, not from task-agnostic policy search. Another possible misconception is that zero-shot RL is already fully solved once a BFM is pretrained. The method’s motivation is precisely that task inference can remain data-hungry if reward labels are scarce or expensive.

6. Scope, assumptions, and terminological disambiguation

The scope of OpTI-BFM is narrower than the phrase “Behavior Foundation Model” might suggest. Its theory is built around linear reward structure, successor features, and exact or near-exact optimization over confidence sets. The assumptions used for the regret result—perfect USFs, linear reward with $z\in\mathcal Z$7-subgaussian noise, an exact optimization oracle, and bounded norms—are stated explicitly as idealized assumptions, and the paper notes that the first two are mainly for theory while the method works empirically even when they are imperfect (Rupf et al., 23 Oct 2025).

The term also requires disambiguation across the broader literature. The data block contains several unrelated uses of BFM and OPTI:

Term in source	Meaning	Source
OpTI-BFM	Optimistic Task Inference for Behavior Foundation Models	(Rupf et al., 23 Oct 2025)
BFM	Behavior Foundation Model for humanoid robots	(Zeng et al., 17 Sep 2025)
PT-BFM	pinch technique–background field method	(Rodríguez-Quintero, 2010)
BFM	branch flow model in optimal power flow	(Geth et al., 2022)
OPTI	optimization-class NLFFF extrapolation code	(Rudenko et al., 2020)

This multiplicity is not merely terminological. In the humanoid-control paper, BFM denotes a generative whole-body control framework trained with masked online distillation and a CVAE; in the OpTI-BFM paper, BFMs are instantiated through successor-feature models used for zero-shot RL. The data explicitly states that the humanoid paper does not use the term “OpTI-BFM.” This suggests that OpTI-BFM is best understood as a later, task-inference-specific development built on the broader BFM idea, rather than as the original name of the behavior model itself.

In summary, OpTI-BFM occupies a specific place in the emerging BFM literature: it addresses the test-time identification of an unseen reward function through interaction, using optimism over latent reward embeddings, least-squares confidence sets, and successor-feature policy retrieval. Its significance lies in showing that BFM-based zero-shot RL need not rely on a large labeled offline inference dataset, while preserving a direct connection to linear bandit theory and retaining low practical overhead.