Optimistic Initial Model in Factored MDPs

• OIM is a reinforcement learning technique that initializes models with inflated reward estimates to encourage thorough exploration in structured state spaces.
• FOIM leverages factored value iteration and per-factor count updates to achieve near-optimal policies with provable polynomial-time and sample efficiency guarantees.
• The method ensures that, except for a polynomial number of steps, actions remain within an epsilon range of the optimal policy as established by rigorous theoretical bounds.

An Optimistic Initial Model (OIM) is a technique for reinforcement learning (RL) in which the agent’s empirical model is initialized with highly optimistic estimates, systematically biasing initial policy selection toward actions that appear maximally rewarding. In the context of factored Markov Decision Processes (FMDPs)—MDPs with structured state representations—FOIM (Factored Optimistic Initial Model) provides the first algorithm that is both “purely greedy” and guarantees polynomial time learning with respect to the fixed point of approximate value iteration (AVI), as established in (0904.3352).

1. Optimistic Initialization in Factored MDPs

FOIM initializes transition and reward factors to embody maximal optimism. For each state variable $X_i$, a fictitious “Garden-of-Eden” (GOE) substate $x_E$ is introduced, expanding each domain to $X_i \cup \{x_E\}$. Empirically, for each local context $z = x[\Gamma_i]$, action $a$, and possible next variable value $y_i$, two counts are maintained: $N_i^{\mathrm{vis}}(z,a)$ (context-action visit count) and $N_i(z,a,y_i)$ (transitions to $y_i$).

Initialization is as follows: $N_i^{\mathrm{vis}}(z, a) = 1$

$x_E$0

This yields empirical transition factors with $x_E$1 initially, so every action-context pair appears to deterministically yield the GOE state. Optimism is similarly infused into rewards: an extra factor $x_E$2 with large constant reward $x_E$3 for $x_E$4 is added,

$x_E$5

Hence, all unexplored regions promise maximal possible future payoff, incentivizing thorough exploration.

2. Algorithmic Structure: FOIM Core Loop

FOIM operates by iteratively updating local empirical models and recomputing the greedy policy. At each step:

1. Empirical Model Update: Recalculate $x_E$6 using the current counts.
2. Factored Value Iteration (FVI): Solve the empirical factored MDP (using FVI) to tolerance $x_E$7, producing new basis weights $x_E$8.
3. Policy Extraction: Greedily select actions,

$x_E$9

1. Interaction: Execute $X_i \cup \{x_E\}$0, observe $X_i \cup \{x_E\}$1.
2. Count Update: For each factor $X_i \cup \{x_E\}$2, increment $X_i \cup \{x_E\}$3 and $X_i \cup \{x_E\}$4.
3. Iterate: Repeat from Step 1.

Each step’s dominant computational operations—the empirical update and planner invocation—incur polynomial cost in all relevant parameters: number of factors $X_i \cup \{x_E\}$5, local table size $X_i \cup \{x_E\}$6, action space size $X_i \cup \{x_E\}$7, number of basis functions $X_i \cup \{x_E\}$8, and accuracy/confidence parameters.

3. Approximate Value Iteration with Factored Structure

FOIM employs Factored Value Iteration (FVI) as its planning subroutine: the agent’s value function is projected onto a basis $X_i \cup \{x_E\}$9 with

$z = x[\Gamma_i]$0

and managed through a normalized projection matrix $z = x[\Gamma_i]$1. FVI applies repeated AVI steps,

$z = x[\Gamma_i]$2

where $z = x[\Gamma_i]$3 is the extended reward vector, $z = x[\Gamma_i]$4 the (Kronecker-factorized) transition operator, and $z = x[\Gamma_i]$5 is the matrix of basis functions. The Bellman backup integrates the optimistic GOE-reward bonus at each step.

4. Polynomial-Time Learning Guarantees

FOIM’s main theoretical result is that, for suitable $z = x[\Gamma_i]$6, with high probability, the number of time steps where the action selected is not $z = x[\Gamma_i]$7-close to the AVI-optimal $z = x[\Gamma_i]$8-value is polynomial in all problem parameters. Specifically, letting $z = x[\Gamma_i]$9 for local-scope size $a$0, and with discount $a$1 and accuracy/confidence parameters $a$2, $a$3,

$a$4

for constant $a$5 ensures,

$a$6

Thus, except for polynomially many steps, FOIM’s policy is nearly as good as obtained by running approximate value iteration on the true model (0904.3352).

5. Outline of the Proof Approach

The convergence argument follows two main pillars:

(a) Persistent Optimism through Initialization and Greediness: When $a$7 is sufficiently large, induction shows that at every iteration

$a$8

The optimism established through initialization persists due to the monotonicity of $a$9 and the per-factor maximally optimistic starting conditions.

(b) Bounded Sample Complexity via Factor-Component Counting: Each local component $y_i$0 is declared $y_i$1-known once visited $y_i$2 times. Using uniform Azuma/Hoeffding arguments, all factor-components (at most $y_i$3) eventually become known, after at most polynomially many visits. A specialized simulation lemma then ensures that once all components are known, the empirical model’s value is within $y_i$4 of the true (AVI) value; further mistakes are rare and coincident only with component discovery.

6. Computational Complexity Analysis

Each FOIM iteration involves:

• Empirical model update: $y_i$5 operations
• Factored Value Iteration: Each Bellman backup is $y_i$6, repeated $y_i$7 times per contraction.

Consequently, total planner time per step is $y_i$8, which is polynomial in all input and accuracy parameters. This renders FOIM, unlike prior approaches, tractable for high-dimensional structured domains.

7. Comparison to Tabular OIM and Algorithmic Implications

Tabular OIM applies the same exploration principle by injecting a GOE pseudo-transition for each global $y_i$9 and employs standard, exact value iteration. FOIM diverges in two respects:

• Factorization of Counts: FOIM maintains per-factor transition counts, reducing storage and computation versus flat $N_i^{\mathrm{vis}}(z,a)$0 tables.
• Use of Approximate Factored Planning: FVI is used in place of exact value iteration, so the performance benchmark is not the true optimum $N_i^{\mathrm{vis}}(z,a)$1, but the AVI fixed point $N_i^{\mathrm{vis}}(z,a)$2.

Despite these differences, the explorative behavior—driven entirely by initial optimism, with no explicit exploration bonus—remains the same. As real experience accrues, optimism fades in locally explored regions, focusing planning on true models. FOIM is distinguished as the first factored-MDP learning algorithm that is greedy, polynomial-time per-iteration, and sample-efficient, staying $N_i^{\mathrm{vis}}(z,a)$3-close to best AVI policy except for a polynomial number of steps (0904.3352).