---
title: 'PAC-ACT: Actor-Critic for Action Chunking'
url: https://www.emergentmind.com/topics/pac-act
type: topic
---

# PAC-ACT: Actor-Critic for Action Chunking

Searching arXiv for PAC-ACT and related Action Chunking Transformer work.
I’m checking arXiv for the PAC-ACT paper and closely related ACT/action-chunking literature.
PAC-ACT denotes **Post-training Actor-Critic for Action Chunking Transformers**, a reinforcement-learning fine-tuning framework for pretrained ACT-style visuomotor policies in precision industrial contact manipulation. It is designed for settings in which low-latency control, low GPU-memory usage, stable contact, and force safety are all operational constraints, and it addresses a specific limitation of behavior-cloned Action Chunking Transformer policies: degradation under distribution shift in contact-rich tasks. PAC-ACT reformulates policy optimization at the chunk level, transfers the pretrained ACT architecture into an actor-critic framework, and adds a hybrid behavior-prior constraint to preserve the pretrained action distribution during online fine-tuning. Reported experiments show improvements in task success, contact stability, and force safety while retaining ACT’s deployment-oriented efficiency; on the Contour task, the proportion of force readings above \(60\,\mathrm N\) decreases by \(46\) times [2607.09590].

## 1. Scope and problem setting

PAC-ACT is situated in **precision contact manipulation** for industrial robotics, where small pose perturbations and contact-force excursions materially affect reliability. The framework targets scenarios in which a robot must maintain stable contact with a workpiece under tight positional tolerances and explicit force limits. In this regime, a policy must recover from small deviations, avoid unsafe force spikes, and sustain long-horizon contact behavior rather than merely reach a nominal pose [2607.09590].

The method is motivated by a contrast between two policy families. Vision-language-action models offer broad generalization, but the cited work characterizes them as incurring high inference latency and GPU-memory cost. Vision-action chunking policies, by contrast, are described as more suitable for real-time industrial control because they produce a future action chunk in a single forward pass and preserve temporal continuity. PAC-ACT takes the latter route, but treats pure behavior cloning as insufficient for contact-rich tasks because compounding error under off-demonstration states can lead either to missed contact or to unsafe interaction forces.

The pretrained ACT configuration used by PAC-ACT employs a **ResNet-18 visual encoder**, a **4-layer Transformer encoder**, and an action decoder that outputs a future sequence of length \(K=100\). The core claim is not that this ACT backbone is inadequate in nominal conditions, but that offline imitation alone does not adequately exploit reward or safety feedback once perturbations accumulate.

## 2. Chunk-level reinforcement-learning formulation

A central technical contribution of PAC-ACT is to align the RL problem with ACT’s native output structure. Standard RL typically optimizes a policy over single-step actions \(a_t\), whereas ACT produces an entire action sequence jointly. PAC-ACT resolves this mismatch by defining one RL decision as one **action chunk** [2607.09590].

If \(c\) environment steps are grouped into one chunk-level decision, the policy observes \(s_\tau\), emits a chunk \(\mathbf a_\tau\), and receives an accumulated chunk reward
\[
R_\tau = \sum_{i=0}^{c-1} r_{\tau c + i},
\]
with inter-chunk discount
\[
\Gamma = \gamma^c.
\]

The chunk-level formulation changes both credit assignment and exploration. The optimization target is a trajectory segment rather than an atomic action, and the paper explicitly treats the actions inside a chunk as temporally coupled rather than independent. Advantage estimation is correspondingly defined at the chunk level:
\[
A_\tau^{\text{GAE}} = \sum_{l=0}^{\infty} (\Gamma\lambda)^l \delta_{\tau+l}, \qquad
\delta_\tau = R_\tau + \Gamma V(s_{\tau+1}) - V(s_\tau).
\]

This construction is significant because it avoids treating ACT as if it were a step-wise policy under a conventional PPO wrapper. Instead, PAC-ACT changes the MDP granularity so that the optimization unit matches the policy’s chunked action generation.

## 3. ACT-transferred actor-critic architecture

PAC-ACT builds both actor and critic directly from the pretrained ACT network. The **actor** preserves the pretrained ACT backbone—visual encoder, Transformer encoder, and action decoder—but removes the **CVAE latent module**. The stated reason is that PPO already injects exploration through a Gaussian policy, so retaining the CVAE would introduce an additional source of stochasticity and create gradient conflict with the baseline constraint. The actor outputs the mean of a diagonal Gaussian:
\[
\mu_\theta(s) = \mathrm{ACT}_{\text{backbone}}(s), \qquad
a \sim \mathcal N(\mu_\theta(s), \Sigma),
\]
with learnable diagonal covariance \(\Sigma\). The reported initialization of the log standard deviation differs by task, including \(-5.5\) for Metal Touch and \(-3.0\) for Square Assembly [2607.09590].

The **critic** reuses the pretrained ACT feature extractor—visual encoder and Transformer encoder—but removes the decoder, since value estimation does not require action generation. The resulting value function is
\[
V(s) = \mathrm{MLP}\big(\mathrm{pool}(\mathrm{ACT}_{\text{encoder}}(s))\big).
\]

The observation space is multimodal, combining **front and wrist RGB images** with **6-DoF joint-angle encoding**. The paper argues that the transferred encoder already contains task-relevant structure, so preserving it while simplifying the critic improves stability. The ablation evidence reported later is consistent with that claim: the encoder-only critic outperforms a decoder-retaining alternative.

## 4. Hybrid behavior-prior constraint and training objective

PAC-ACT regularizes RL fine-tuning with what the paper terms a **hybrid behavior-prior constraint**. This has two components: a KL penalty between successive policies and a reward-level penalty that keeps the fine-tuned policy close to a frozen pretrained ACT baseline \(\pi_{\text{base}}\) [2607.09590].

The training loss includes a trust-region style KL term,
\[
\mathcal L_{\text{train}}
=
-\mathcal L_{\text{PPO}}
+
\beta_1 \,\widehat D_{\text{KL}}(\pi_{\theta_{\text{old}}}\|\pi_\theta)
+\ldots
\]
with \(\beta_1 = 3.0\). This term constrains distributional drift between PPO updates, which is particularly relevant because action chunks are high-dimensional.

The second component modifies the rollout reward:
\[
r'_t
=
r_t
-
\beta_2 \,\|\mathbf a_\theta(s_t)-\mathbf a_{\text{base}}(s_t)\|^2,
\]
with \(\beta_2 = 2.0\). This penalty keeps exploration near the pretrained ACT action manifold, which the paper presents as essential in sparse-reward or high-dimensional chunk spaces where unconstrained exploration can drift into unsupported behaviors.

The PPO objective is written over chunk actions:
\[
\mathcal L_{\text{PPO}}
=
\mathbb E_t\Big[
\min\big(
r_t(\theta)\hat A_t,\;
\mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat A_t
\big)
\Big],
\]
where the probability ratio is
\[
\rho_t(\theta)
=
\frac{\pi_\theta(\mathbf a_t\mid s_t)}
{\pi_{\theta_{\text{old}}}(\mathbf a_t\mid s_t)}.
\]

The full loss combines the negative PPO objective, the KL term, critic regression, and entropy regularization:
\[
\mathcal L
=
-\mathcal L_{\text{PPO}}
+
\beta_1 \widehat D_{\text{KL}}(\pi_{\theta_{\text{old}}}\|\pi_\theta)
+
c_v \mathcal L_{\text{value}}
-
c_e \mathcal H(\pi_\theta).
\]

For the Metal Touch environment, the dense reward is
\[
\begin{split}
r_t ={}& r_{\text{touch}} - p_{\text{dist}} - p_{\text{smooth}} - p_{\text{orient}} \\
& - p_{\text{time}} - p_{\text{force}} + r_{\text{success}}.
\end{split}
\]
The force penalty is tied to the total force magnitude,
\[
f_{\text{total}} = \sqrt{f_x^2 + f_y^2 + f_z^2}.
\]

A plausible implication is that PAC-ACT’s distinctive contribution lies less in PPO itself than in how PPO is constrained so that online RL improves safety and robustness without erasing the pretrained ACT behavior.

## 5. Training pipeline and benchmark environments

The training pipeline begins with **offline ACT pretraining** on expert demonstrations in the LeRobot framework; the main model uses **800 expert trajectories**. PAC-ACT then initializes the actor from the pretrained ACT, initializes the critic from the ACT encoder, freezes the baseline ACT policy for the reward penalty, and performs online chunk-level PPO fine-tuning [2607.09590].

The primary benchmark is **Metal Touch**, a MuJoCo precision-contact environment in which a 6-DoF arm with a probe-like end-effector must touch designated regions on a rubber-metal component. To model industrial uncertainty, the object is randomly shifted by up to \(\pm 2\) cm per episode. Three tasks are emphasized:

- **Diamond**: touching four side points in sequence.
- **Cross**: repeated center crossings with observation-action ambiguity.
- **Contour**: continuous motion along the inner boundary of the ring; this is described as the hardest and most contact-intensive task.

The policy observes **front and wrist RGB images** and **6 joint angles**, while force/torque readings are available in simulation for training and evaluation. The control frequency is **10 Hz**.

PAC-ACT is also evaluated on **Square Assembly** from robomimic as a transfer and scalability test beyond Metal Touch. Baseline comparisons include ACT, Diffusion Policy, and the vision-language-action baseline \(\pi\)0.5.

## 6. Reported performance, ablations, and limitations

The reported empirical results emphasize three properties simultaneously: **task success**, **completion efficiency**, and **force safety** [2607.09590]. On the **Contour** task, success rates are reported as **60.0%** for ACT, **60.0%** for Diffusion Policy, **79.0%** for \(\pi\)0.5, and **100.0%** for PAC-ACT. On **Square Assembly**, the corresponding values are **51.2%**, **77.8%**, **62.6%**, and **98.2%**.

Completion efficiency improves as well. On Contour, average completion steps decrease from **485.4** to **170.6**, and episode duration decreases from **48.5 s** to **17.1 s** after fine-tuning. The system also retains ACT’s deployment profile: **mean inference latency 88.1 ms**, **P95 latency 125.6 ms**, and **GPU memory 2.30 GB**. The cited comparison point for \(\pi\)0.5 is **44.15 GB** peak GPU memory.

Force-safety measurements are particularly prominent. Relative to ACT on Contour, PAC-ACT reduces **median peak force** from **105.40 N** to **20.74 N**, **mean force** from **\(38.67 \pm 94.12\) N** to **\(3.92 \pm 4.31\) N**, and **maximum peak force** from **8452.5 N** to **120.9 N**. The **proportion of force readings above \(60\,\mathrm N\)** decreases from **4.6%** to **0.1%**, and the **fraction of episodes with any reading above \(60\,\mathrm N\)** decreases from **20%** to **6%**. The abstract summarizes the first of these effects as a **46 times** reduction in the proportion of readings above the \(60\,\mathrm N\) threshold.

The ablation study attributes these gains to specific architectural decisions. A decoder-free critic attains **\(96.4\% \pm 3.0\%\)** final-window mean success rate, compared with **\(88.0\% \pm 7.4\%\)** for a critic that retains the Transformer decoder. An actor that removes the CVAE reaches **\(96.4\% \pm 3.0\%\)**, whereas the CVAE-retained actor reaches **\(83.1\% \pm 7.8\%\)**. In the sparse-reward setting, retaining the KL behavior-prior constraint yields **100% rollout success** and preserves structured contour-following behavior under randomized initial poses; removing that constraint leads to **22.0%** cumulative total success and a final rollout window of **72.7%**. This suggests that success-rate statistics alone can obscure degradation in trajectory structure when the behavior prior is absent.

The reported limitations are also explicit. The evaluation is **mostly simulation-based**; sim-to-real transfer is not yet established. The perturbations are mainly randomized object positions rather than broader changes such as lighting, viewpoint, or dynamics. The strongest results use **dense rewards**, although sparse-reward ablations are included. The removal of the CVAE may simplify optimization at the cost of reduced multimodality, and the authors present the method as oriented toward **industrial precision-contact control** rather than as a universal policy-learning framework.

In that sense, PAC-ACT is best understood as a deployment-focused post-training method: it preserves the low-latency, low-memory, temporally continuous structure of Action Chunking Transformers, while using chunk-level actor-critic optimization and explicit behavior priors to improve robustness and force-constrained safety in contact-rich industrial manipulation.

Source: https://www.emergentmind.com/topics/pac-act