---
title: Model-Based RL with Action Chunks (MAC)
url: https://www.emergentmind.com/topics/model-based-rl-with-action-chunks-mac
type: topic
---

# Model-Based RL with Action Chunks (MAC)

Searching arXiv for the MAC paper and closely related action-chunking references to ground the article.
Using the arXiv search tool to verify the cited papers and surrounding literature.
Model-Based RL with Action Chunks (MAC) denotes a form of offline model-based reinforcement learning in which the learned predictive interface is lifted from primitive actions to fixed-length sequences of actions. In the formulation introduced in "Scalable Offline Model-Based RL with Action Chunks" [2512.08108], the dynamics model predicts a future state from a current state and an action chunk, model-based value expansion is performed over chunk-level imagined rollouts, and policy extraction is carried out by rejection sampling from an expressive behavioral action-chunk policy. The method is motivated by a specific long-horizon trade-off in offline model-based RL: larger value-expansion horizons reduce bootstrap bias, but one-step autoregressive world models accumulate model error over long rollouts [2512.08108].

## 1. Conceptual definition and scope

In MAC, an action chunk is a fixed-length sequence of primitive actions, written as \(a_{i:j} = (a_i, a_{i+1}, \dots, a_j)\). The core chunked transition interface is
\[
p(s_{t+n}\mid s_t, a_{t:t+n-1}),
\]
with a matching chunk policy
\[
\pi(a_{t:t+n-1}\mid s_t).
\]
Relative to standard one-step model-based RL, the key change is that one model call now advances the imagined trajectory by \(n\) environment steps rather than one [2512.08108].

This gives MAC a specific position within the broader literature on action abstraction. It is directly about temporal abstraction, because the learned model consumes a sequence of primitive actions and predicts a state \(n\) steps later. It is not merely an action reparameterization. This distinguishes it from "Predictable MDP Abstraction for Unsupervised Model-Based RL" [2302.03921], which learns a latent action space for model-based control but explicitly frames its decoder as permitting predictable actions “without temporal abstraction” [2302.03921]. It is also distinct from model-free chunked RL methods such as "Reinforcement Learning with Action Chunking" [2507.07969], which run TD learning directly in a chunked action space but do not learn a dynamics model [2507.07969].

The broader MAC concept therefore refers to model-based RL methods that reason over temporally extended action sequences as the operative control unit. The named MAC recipe of [2512.08108] is the most explicit instantiation in the provided literature: an offline model-based actor-critic with a chunked dynamics model, a chunked reward model, long-horizon chunk-based value expansion, and critic-guided rejection sampling from a behavioral chunk prior.

## 2. Core formulation and algorithmic recipe

The starting point is the offline RL setting in the MDP
\[
\mathcal{M} = (\mathcal{S}, \mathcal{A}, r, p, \mu),
\]
with a fixed offline dataset \(\mathcal{D}\) and no further environment interaction [2512.08108]. MAC learns two world-model components from chunked transitions \((s_t, a_{t:t+n-1}, r_t, s_{t+n})\): a chunked dynamics model
\[
p_\psi(s_t,a_t) \approx s_{t+n},
\]
and a chunked reward model
\[
r_\phi(s_t,a_t) \approx \sum_{i=0}^{n-1}\gamma^i r_{t+i},
\]
where \(a_t\) is shorthand for the chunk \(a_{t:t+n-1}\) [2512.08108]. Their training losses are
\[
\mathcal{L}_{\mathrm{dyn}}(\psi)=\mathbb{E}\!\left[\|p_\psi(s_t,a_t)-s_{t+n}\|_2^2\right],
\qquad
\mathcal{L}_{\mathrm{rew}}(\phi)=\mathbb{E}\!\left[\|r_\phi(s_t,a_t)-r_t\|_2^2\right].
\]

The policy side is not trained as a conventional reward-maximizing actor. Instead, MAC fits an expressive behavioral action-chunk policy by flow matching and then distills that ODE-based sampler into a one-step MLP sampler for efficiency [2512.08108]. At decision time, the policy is defined distributionally by rejection sampling:
\[
\pi(s_t)\overset{d}{=}\arg\max_{\{a_{t:t+n-1}^{\,i}\}_{i=1}^N\sim \pi_B(\cdot\mid s_t)} Q(s_t,a_{t:t+n-1}),
\]
so candidate chunks come from the behavioral model, while the critic chooses among them [2512.08108]. This is a central design choice: MAC controls out-of-distribution action selection by restricting the candidate set to BC-generated chunk proposals rather than relying primarily on explicit uncertainty penalties.

Value learning uses chunk-based model-based value expansion. Starting from real dataset states, MAC rolls out the rejection-sampling policy in the chunked world model for \(H\) chunk steps, producing imagined trajectories that span \(nH\) primitive environment steps:
\[
\mathcal{D}_{\mathrm{img}}=
\left\{
\left(
s_t^{(i)},\hat a_t^{(i)},\hat r_t^{(i)},\hat s_{t+n}^{(i)},\hat a_{t+n}^{(i)},\ldots,\hat s_{t+Hn}^{(i)}
\right)
\right\}_{i=1}^M.
\]
The state-value loss is then
\[
\mathcal{L}_V(\psi)=
\mathbb{E}\!\left[
\left(
V_\psi(s_{t+kn})
-
\sum_{i=k}^{H-1}\gamma^{(i-k)n}\hat r_{t+in}
-
\gamma^{(H-k)n}V_{\bar\psi}(\hat s_{t+Hn})
\right)^2
\right],
\]
while the chunk critic is trained with
\[
\mathcal{L}_Q(\psi)=
\mathbb{E}\!\left[
\left(
Q_\psi(s_t,\hat a_t)-\hat r_t-\gamma^n[V_{\bar\psi}(\hat s_{t+n})]_{\times}
\right)^2
\right].
\]
The practical implication is that MAC performs value expansion over long primitive horizons while only recursively applying the dynamics model at chunk resolution [2512.08108].

This reduction in recursive model depth is the main technical rationale for the method. A one-step model must be queried \(nH\) times to predict \(nH\) primitive steps. MAC instead needs only \(H\) recursive model transitions, because each model call predicts the state after an entire chunk. The paper argues that this changes the usual model-based value-expansion trade-off by allowing long-horizon imagined returns without equally long autoregressive model chains [2512.08108].

## 3. Relation to earlier and adjacent action abstractions

MAC sits at the intersection of two earlier lines of work: learned action abstractions for model-based control, and chunked action spaces for long-horizon RL. The most directly relevant model-free precursor is Q-chunking, introduced in "Reinforcement Learning with Action Chunking" [2507.07969]. Q-chunking treats a fixed-length open-loop action sequence as the effective RL action, trains a chunk-level critic \(Q(s_t,a_{t:t+h})\), and argues that chunk-valued backups can provide unbiased multi-step TD targets because the critic conditions on the exact executed sequence [2507.07969]. MAC inherits the same basic insight—that long-horizon learning can be stabilized by elevating the control unit from a primitive action to a short open-loop sequence—but moves that idea into an explicitly model-based value-expansion setting [2512.08108].

A different adjacent line appears in "Predictable MDP Abstraction for Unsupervised Model-Based RL" [2302.03921]. PMA is genuinely model-based and learns a latent action space used for downstream planning, but its latent actions are decoded into single primitive actions at the current step and the paper explicitly contrasts its goal with temporally extended skill learning, describing the method as action transformation “without temporal abstraction” [2302.03921]. PMA is therefore relevant to MAC as an example of learned abstract actions for model-based control, but not as an action-chunk method in the temporal sense.

Another neighboring direction is demonstration-derived chunk discovery or chunk regularization without a world model. "Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization" [2511.15055] distills human demonstrations into fixed-length macro actions via a conditional VQ-VAE and motivates the method with receding-horizon control, but the implementation does not learn a dynamics model and the online decision rule is best characterized as RL over a learned discrete macro-action space [2511.15055]. Likewise, "Action abstractions for amortized sampling" [2410.15184] mines frequent action subsequences from successful trajectories and inserts them into the action vocabulary, but it does so in model-free RL and GFlowNet settings rather than in model-based planning [2410.15184].

These comparisons delimit MAC precisely. It is not simply any action abstraction for model-based control, and it is not simply any chunked RL method. Its distinctive identity comes from combining a learned chunk-transition model, chunk-level reward prediction, model-based value expansion, and behavioral rejection sampling in offline RL [2512.08108].

## 4. Empirical profile of the MAC recipe

The empirical focus of MAC is long-horizon offline RL on large datasets. On OGBench-derived goal-conditioned tasks with datasets of up to 100M transitions, the paper reports that MAC achieves the best performance among offline model-based RL algorithms, especially on challenging long-horizon tasks [2512.08108]. Representative results include cube-double at \(100 \pm 1\), cube-octuple at \(30 \pm 6\), puzzle-3x3 at \(100 \pm 0\), and puzzle-4x5 at \(99 \pm 3\). In the same table, F-MPC reports \(0 \pm 0\) on cube-octuple and \(0 \pm 0\) on puzzle-4x5, illustrating the gap on the hardest manipulation domains [2512.08108].

On standard reward-based manipulation benchmarks, MAC reports averages of \(99 \pm 2\) on cube-single, \(53 \pm 4\) on cube-double, \(97 \pm 4\) on scene, \(20 \pm 0\) on puzzle-3x3, and \(78 \pm 13\) on puzzle-4x4, and the paper states that it achieves the best performance on 4/5 environments [2512.08108]. The stronger performance is concentrated in longer-horizon manipulation tasks, while humanoid locomotion-style environments remain difficult for all model-based methods considered.

The ablation results clarify what the chunk abstraction is buying. When chunk length is varied over \(n\in\{1,5,10,25\}\), larger chunk sizes substantially reduce rollout model error, and the one-step model diverges over long rollouts [2512.08108]. At the same time, the policy-performance ablation shows that chunking helps only up to a point: on cube-octuple, no chunking (\(n=1\)) cannot solve the task at all, but excessively large chunks (\(n=25\)) hurt because open-loop prediction and chunk action-value estimation both become harder [2512.08108]. The intended conclusion is not that “larger is always better,” but that chunking moves the bias/model-error trade-off into a more favorable regime when the chunk size is chosen appropriately.

The behavioral prior is equally central. Replacing the flow-based chunk policy with a Gaussian policy causes severe collapse: MAC(Gau) reports \(2 \pm 3\) on cube-single, \(0 \pm 0\) on cube-double, \(0 \pm 0\) on scene, and \(0 \pm 0\) on puzzle-4x4, versus \(100 \pm 0\), \(50 \pm 12\), \(100 \pm 0\), and \(85 \pm 14\) for the full MAC configuration in the same ablation table [2512.08108]. Distillation from the ODE flow sampler into the one-step sampler is also not optional: training the fast sampler directly with BC instead of distilling from the flow model yields major failure across the same tasks [2512.08108]. These results indicate that MAC depends not only on chunked dynamics, but also on an expressive, multi-modal chunk prior that keeps policy extraction on the support of the offline data.

Implementation details reinforce the intended scale. The default configuration uses four-layer MLPs with layer normalization and GELU, learning rate \(3\times 10^{-4}\), target update rate \(0.005\), discount factor \(\gamma=0.999\), action chunk size \(n=10\), and rollout length \(H=10\), so the value target spans roughly 100 environment steps [2512.08108]. The same \((n,H)=(10,10)\) is used across all tasks in the main setup.

## 5. Adaptive duration, execution, and reactive control around MAC

Subsequent adjacent work has concentrated less on building new chunked world models than on three unresolved issues that MAC leaves open: how long a chunk should be, how chunk execution should interact with feedback, and whether chunk reasoning must imply chunk commitment at test time.

One response is adaptive chunk-length selection. "Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning" [2605.05544] argues that fixed chunk size is structurally mismatched to robotics tasks and proposes selecting among several chunk lengths by comparing a per-horizon, discount-normalized advantage rather than raw \(Q\)-values [2605.05544]. "ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network" [2605.11009] and "Adaptive Action Chunking via Multi-Chunk Q Value Estimation" [2605.10044] push this further with causal Transformer critics that score every prefix of a proposed chunk and choose the best execution length at each chunk boundary, again without a learned dynamics model [2605.11009][2605.10044]. For MAC, these papers suggest that fixed \(n\) is likely a simplifying assumption rather than a fundamental requirement.

A second line concerns reactivity under chunked execution. "SEAR: Sample Efficient Action Chunking Reinforcement Learning" [2603.01891] combines large chunk sizes with receding-horizon execution by training chunk policies of size \(N\) but collecting data with random replanning intervals \(k\sim\mathrm{Uniform}(1,N)\), and reports that training with larger chunks and evaluating with shorter replanning intervals can outperform training directly at the shorter chunk size [2603.01891]. "Temporal Action Selection for Action Chunking" [2511.04421] addresses a related problem at inference time by selecting among overlapping chunk proposals generated at different timesteps, thereby trying to recover both reactivity and motion coherence without abandoning chunk structure [2511.04421]. The common implication is that open-loop chunk commitment and chunk-based reasoning need not coincide.

A third line concerns training-time use of chunk abstractions without chunk-level execution at deployment. "Chunk-Guided Q-Learning" [2603.13971] trains an auxiliary chunk critic with temporally extended backups, but regularizes a single-step critic toward that chunk critic and returns a single-step policy at test time [2603.13971]. This is conceptually significant for MAC because it shows a distinct design philosophy: plan or learn with chunks, act reactively.

Real-time deployment work sharpens the execution problem further. "Real-Time Robot Execution with Masked Action Chunking" [2601.20130] studies asynchronous inference for chunked policies and argues that failures arise not only from inter-chunk discontinuity but also from intra-chunk inconsistency, where the executed prefix is stale relative to current perception [2601.20130]. Although REMAC is not model-based, its diagnosis is directly relevant to any MAC system intended for robotics deployment.

## 6. Limitations, controversies, and open questions

The limitations of MAC are stated plainly in its own experiments. The method remains weak on long-horizon humanoid locomotion tasks, and the paper explicitly identifies contact-rich, erratic dynamics as a remaining bottleneck for model-based methods [2512.08108]. The world model in the main recipe is a deterministic MLP, and the authors suggest that more expressive generative or latent dynamics models are a natural future direction [2512.08108]. A plausible implication is that chunking alone does not remove the need for stronger world models when dynamics are highly discontinuous or partially observed.

A second limitation is that the chunk size is fixed. MAC uses the same \((n,H)=(10,10)\) across tasks, and its ablations show both that chunking is essential and that excessively large chunks become difficult to evaluate and execute [2512.08108]. Later adaptive-length methods make this limitation explicit by arguing that the optimal chunk size varies across both tasks and states [2605.05544][2605.11009][2605.10044]. This suggests that fixed chunk duration is one of the main remaining simplifications in the original MAC formulation.

A third limitation concerns feedback. MAC’s value expansion and policy extraction are chunk-centric, but the paper does not develop a detailed receding-horizon execution scheme in the style of MPC. Adjacent work on chunked control repeatedly emphasizes that long chunks reduce responsiveness, that asynchronous or delayed execution introduces train-test mismatch, and that execution-time arbitration among chunk prefixes or overlapping chunk proposals can materially affect performance [2601.20130][2511.04421][2603.01891]. This suggests that the full MAC problem is not exhausted by learning \(p(s_{t+n}\mid s_t,a_{t:t+n-1})\); it also includes deciding how much of a planned chunk to commit before replanning.

Finally, there is a broader conceptual controversy over what chunking is for. In MAC, chunking is primarily a device for reducing compounding model error and supporting long-horizon value expansion [2512.08108]. In Q-chunking, it is also a device for unbiased multi-step TD backups and temporally coherent exploration [2507.07969]. In MAQ, it becomes a mechanism for human-like behavior regularization [2511.15055]. In REMAC and TAS, the focus shifts to execution reliability and reactivity [2601.20130][2511.04421]. The literature therefore does not support a single universal interpretation of action chunks. Instead, it shows that the same abstraction can serve different roles: model simplification, search-space restriction, behavior prior, temporal smoothing, or real-time systems support.

In that broader sense, MAC is best understood as the model-based member of a larger action-chunking family. Its specific contribution is to show that chunked world models and chunk-restricted policy extraction can make model-based value expansion scale to much longer offline horizons than one-step autoregressive alternatives [2512.08108]. Its unresolved questions—adaptive duration, reactive execution, stronger world models, and uncertainty-aware chunk planning—define much of the surrounding research agenda.

Source: https://www.emergentmind.com/topics/model-based-rl-with-action-chunks-mac