---
title: Variational Markovian Option Critic (VMOC)
url: https://www.emergentmind.com/topics/variational-markovian-option-critic-vmoc
type: topic
---

# Variational Markovian Option Critic (VMOC)

The Variational Markovian Option Critic (VMOC) is an off-policy hierarchical reinforcement learning (HRL) algorithm founded on variational inference within the HiT-MDP (Hierarchical, Temporally-Extended Markov Decision Process) framework. It is designed to construct a diverse library of latent, temporally-extended actions (“options”) as abstract skill embeddings, supporting high-level reasoning in both language and control tasks. VMOC enables implicit reasoning by learning and operating in an abstract latent option space, circumventing the computational cost of explicit, step-by-step reasoning traces, with rigorous guarantees on policy optimality via continuous MDP homomorphisms [2507.16473].

## 1. Probabilistic Graphical Model and Latent Option Embeddings

VMOC’s framework introduces two binary “optimality” variables at each time step: $\mathcal{A}^{(A)}_t \in \{0,1\}$ for actions and $\mathcal{O}^{(O)}_t \in \{0,1\}$ for options. A full-option trajectory takes the form $\tau = \{s_0, o_{-1}, a_0, o_0, s_1, a_1, o_1, \ldots \}$.

The joint optimal-trajectory distribution is:

$$
P(\tau, \mathcal{A}_{1:T}, \mathcal{O}_{1:T}) \propto P(s_0) \prod_{t=0}^{T-1} P(s_{t+1}|s_t,a_t)\; P(\mathcal{A}_t|s_t,a_t)\; P(\mathcal{O}_t|s_t,a_t,o_{t-1})
$$

with the local likelihoods:

- $P(\mathcal{A}_t = 1 | s_t, a_t) = \exp[r(s_t, a_t)]$, where $r$ is environmental reward,
- $P(\mathcal{O}_t = 1 | s_t, a_t, o_{t-1}) = \exp[f(s_t, a_t, o_{t-1})]$, where $f$ is a non-positive diversity regularizer, instantiated as mutual information $f = I[O; (s,a)]$.

The true trajectory posterior is approximated by the HiT-MDP variational distribution $q$:

$$
q(\tau) = P(s_0) \prod_{t=0}^{T-1} P(s_{t+1}|s_t, a_t) \; \pi^A(a_t|s_t, o_t) \; \pi^O(o_t|s_t, o_{t-1})
$$

- $\pi^A$ is the intra-option policy,
- $\pi^O$ is the option policy, implemented via a learned embedding matrix $W\in\mathbb{R}^{K \times d}$: $o_t \sim \mathrm{Categorical}(\mathrm{softmax}(W\cdot\phi(s_t,o_{t-1})))$.
- Option variables $o_t$ function as discrete latent “thought” embeddings.

This graphical model enables abstraction of decision-making into a space of diverse latent skills.

## 2. Variational Objective and ELBO Derivation

The central optimization target is the evidence lower bound (ELBO) on the likelihood of optimal trajectories:

$$
\mathcal{L}(q) = \mathbb{E}_{q(\tau)}\left[ \log P(\tau, \mathcal{A}, \mathcal{O}) - \log q(\tau) \right]
$$

Upon substitution and cancellation of model dynamics terms:

$$
\mathcal{L}(q) = \mathbb{E}_{q} \left[ \sum_t \left(r(s_t, a_t) + f(s_t, a_t, o_{t-1}) - \log \pi^A(a_t|s_t, o_t) - \log \pi^O(o_t|s_t, o_{t-1}) \right) \right]
$$

Grouping yields expected reward plus diversity against the joint entropic regularization of both policies:

$$
\mathcal{L}(q) = \mathbb{E}_q\left[ \sum_t (r + f) \right] + \mathbb{E}_q\left[\sum_t H(\pi^A(\cdot|s_t,o_t)) + H(\pi^O(\cdot|s_t,o_{t-1})) \right]
$$

Maximizing $\mathcal{L}(q)$ thus trades off task reward, option diversity, and policy entropy, directly paralleling the soft option-critic principle.

## 3. Continuous HiT-MDP Homomorphisms and Optimality Guarantees

Let state-option pairs be $e = (s, o)$, forming the associated vector bundle $E_O \to S$. An abstract MDP $\overline{\mathcal{D}}$ arises via bundle map $f: E_O \to \overline{E}_{\bar O}$ and action map $g_e: (a,o) \mapsto (\bar a, \bar o)$, maintaining:

- **Reward invariance:** $R(e, (a,o)) = \overline{R}(f(e), g_e(a,o))$,
- **Transition equivariance:** $\tau_{(a,o)}(f^{-1}(B)|e) = \overline{\tau}_{g_e(a,o)}(B|f(e))$.

**Optimal Value Equivalence Theorem:** If $h = (f, g)$ is a continuous HiT-MDP homomorphism,

$$
Q^*(e,a,o) = \overline{Q}^*(f(e),g_e(a,o))
$$

**Policy Lifting** asserts that, for an abstract policy $\bar{\pi}(\bar a, \bar o | f(e))$, a lifted policy $\pi^{\uparrow}(\cdot|e)$ exists such that its pushforward under $g_e$ equals $\bar{\pi}$, ensuring

$$
Q^{\pi^{\uparrow}}(e,a,o) = \overline{Q}^{\bar{\pi}}(f(e), g_e(a,o))
$$

Hence, solving the abstract MDP via VMOC recovers optimality in the original, unabstracted MDP, with guarantees stemming from locally compact Hausdorff action–option space representation theory.

## 4. The Off-Policy VMOC Algorithm and Training Procedure

VMOC operationalizes the ELBO maximization with an off-policy actor–critic architecture. Core components:

| Component Type | Name/Notation    | Role                        |
|:---------------|:----------------|:----------------------------|
| Critic         | $\psi^A$, $\psi^O$ | $Q^A(s,o,a)$ and $Q^O(s,o)$ |
| Policy         | $\theta^A$, $\theta^O$ | $\pi^A(a|s,o)$ and $\pi^O(o|s,o_-)$ |
| Temperature    | $\alpha^A$, $\alpha^O$ | Regulation of policy entropy |

Training proceeds via environment rollouts and experience replay. Key steps:

1. Collect transitions $(s_t, o_{t-1}, a_t, o_t, r_t, s_{t+1}, o_t)$ into buffer $D$.
2. Sample batches for updates.
3. Optimize critic targets:

$$
y^A_t = r_t + \gamma \mathbb{E}_{o'\sim\pi^O}\left[ Q^O_{\bar \psi^O}(s_{t+1}, o') - \alpha^O \log \pi^O(o'|s_{t+1}, o_t) \right]
$$

$$
y^O_t = f(s_t, a_t, o_{t-1}) + \mathbb{E}_{a' \sim \pi^A}\left[ Q^A_{\bar \psi^A}(s_t, o_t, a') - \alpha^A \log \pi^A(a'|s_t, o_t) \right]
$$

4. Policy update via entropy-regularized gradients:

$$
J_{\pi^A} = -\mathbb{E}_D\left[ \min_i Q^A_{\psi_i}(s_t, o_t, \tilde a_t) - \alpha^A \log \pi^A(\tilde a_t|s_t, o_t)\right]
$$

$$
J_{\pi^O} = -\mathbb{E}_D\left[ \min_i Q^O_{\psi_i}(s_t, o_{t-1}, \tilde o_t) - \alpha^O \log \pi^O(\tilde o_t|s_t, o_{t-1})\right]
$$

5. Entropy temperature tuning:

$$
J(\alpha^A) = \mathbb{E}_D\left[-\alpha^A(\log \pi^A(a_t|s_t,o_t) + \bar H)\right]
$$

$$
J(\alpha^O) = \mathbb{E}_D\left[-\alpha^O(\log \pi^O(o_t|s_t,o_{t-1}) + \bar H)\right]
$$

Adam optimizer and soft updates for target networks are used throughout.

## 5. Core Update Equations and Hyper-Parameter Specifications

VMOC employs $\ell_2$ critic losses and policy gradients derived as $\nabla_{\theta} \mathbb{E}[Q - \alpha \log \pi]$. Default hyper-parameters as outlined:

| Parameter                   | Value                   | Description                |
|:----------------------------|:------------------------|:---------------------------|
| Discount ($\gamma$)         | 0.99                    | Future reward discount      |
| Critic-target ($\tau$)      | 0.005                   | Polyak averaging parameter  |
| Batch size                  | 256                     | For replay buffer sampling  |
| Replay buffer size           | $10^{6}$                 | Experience replay capacity  |
| Learning rate (actor/critic/temp) | $3 \times 10^{-4}$ | Adam learning rates         |
| Adam $\epsilon$             | $10^{-5}$               | Adam optimizer epsilon      |
| Initial $\alpha$’s          | 0.1                     | Entropy regularization      |
| Target entropies ($\bar H$) | $-|A|$, $-|O|$           | For actions, options        |
| Latent options ($K$)        | 4                       | Number of option embeddings |
| Option embedding dim ($d$)  | 40                      | Embedding vector dimension  |

These settings support robust convergence and diversity in the learned option set.

## 6. Theoretical Guarantees and Optimality Preservation

Two main theoretical results underpin VMOC’s guarantee of global optimality:

- **Soft Option Policy Iteration (Theorem 3.2):** Under exact tabular inference, iterated policy evaluation (E-step) and policy improvement (M-step) over $\mathcal{L}(q)$ converges to the unique ELBO maximizer, aligned with the true optimal soft option policies $(\pi^{A*}, \pi^{O*})$.

- **Continuous HiT-MDP Homomorphism (Theorems 5.1 & 5.4):** If $h = (f, g)$ defines a homomorphic abstraction, then $Q^*(e, a, o) = \overline{Q}^*(f(e), g_e(a, o))$, and any lifted policy $\pi^{\uparrow}$ of the abstract optimum achieves identical value in the original MDP.

Together, these results imply that VMOC’s variational inference maximizes a principled lower bound of the control objective, while the abstract optimal policy discovered in latent option space lifts to a true optimum in the original process, with no loss of optimality.

## 7. Applications and Empirical Validation

VMOC’s approach has been validated on both complex logical reasoning benchmarks and challenging locomotion domains. The learned options serve as latent “reasoning steps” in language settings and as abstract skills in control environments, offering an efficient, implicit alternative to explicit step-by-step reasoning such as Chain-of-Thought prompting. The framework’s combination of temporal abstraction, diversity regularization, and strong theoretical grounding supports its deployment for scalable, robust skill acquisition across modalities [2507.16473].

Source: https://www.emergentmind.com/topics/variational-markovian-option-critic-vmoc