PAC-ACT: Actor-Critic for Action Chunking

• PAC-ACT is a reinforcement-learning framework that refines pretrained Action Chunking Transformer policies for precision industrial contact manipulation.
• It reformulates policy optimization at the chunk level and transfers the pretrained architecture into an actor-critic setup while removing the CVAE module.
• Experiments demonstrate significant gains in task success, efficiency, and force safety, including a 46-times reduction in excessive force readings.

Searching arXiv for PAC-ACT and related Action Chunking Transformer work. I’m checking arXiv for the PAC-ACT paper and closely related ACT/action-chunking literature. PAC-ACT denotes Post-training Actor-Critic for Action Chunking Transformers, a reinforcement-learning fine-tuning framework for pretrained ACT-style visuomotor policies in precision industrial contact manipulation. It is designed for settings in which low-latency control, low GPU-memory usage, stable contact, and force safety are all operational constraints, and it addresses a specific limitation of behavior-cloned Action Chunking Transformer policies: degradation under distribution shift in contact-rich tasks. PAC-ACT reformulates policy optimization at the chunk level, transfers the pretrained ACT architecture into an actor-critic framework, and adds a hybrid behavior-prior constraint to preserve the pretrained action distribution during online fine-tuning. Reported experiments show improvements in task success, contact stability, and force safety while retaining ACT’s deployment-oriented efficiency; on the Contour task, the proportion of force readings above $60\,\mathrm N$ decreases by $46$ times (Pang et al., 10 Jul 2026).

1. Scope and problem setting

PAC-ACT is situated in precision contact manipulation for industrial robotics, where small pose perturbations and contact-force excursions materially affect reliability. The framework targets scenarios in which a robot must maintain stable contact with a workpiece under tight positional tolerances and explicit force limits. In this regime, a policy must recover from small deviations, avoid unsafe force spikes, and sustain long-horizon contact behavior rather than merely reach a nominal pose (Pang et al., 10 Jul 2026).

The method is motivated by a contrast between two policy families. Vision-language-action models offer broad generalization, but the cited work characterizes them as incurring high inference latency and GPU-memory cost. Vision-action chunking policies, by contrast, are described as more suitable for real-time industrial control because they produce a future action chunk in a single forward pass and preserve temporal continuity. PAC-ACT takes the latter route, but treats pure behavior cloning as insufficient for contact-rich tasks because compounding error under off-demonstration states can lead either to missed contact or to unsafe interaction forces.

The pretrained ACT configuration used by PAC-ACT employs a ResNet-18 visual encoder, a 4-layer Transformer encoder, and an action decoder that outputs a future sequence of length $K=100$. The core claim is not that this ACT backbone is inadequate in nominal conditions, but that offline imitation alone does not adequately exploit reward or safety feedback once perturbations accumulate.

2. Chunk-level reinforcement-learning formulation

A central technical contribution of PAC-ACT is to align the RL problem with ACT’s native output structure. Standard RL typically optimizes a policy over single-step actions $a_t$, whereas ACT produces an entire action sequence jointly. PAC-ACT resolves this mismatch by defining one RL decision as one action chunk (Pang et al., 10 Jul 2026).

If $c$ environment steps are grouped into one chunk-level decision, the policy observes $s_\tau$, emits a chunk $\mathbf a_\tau$, and receives an accumulated chunk reward

$R_\tau = \sum_{i=0}^{c-1} r_{\tau c + i},$

with inter-chunk discount

$\Gamma = \gamma^c.$

The chunk-level formulation changes both credit assignment and exploration. The optimization target is a trajectory segment rather than an atomic action, and the paper explicitly treats the actions inside a chunk as temporally coupled rather than independent. Advantage estimation is correspondingly defined at the chunk level: $$A_\tau^{\text{GAE}} = \sum_{l=0}^{\infty} (\Gamma\lambda)^l \delta_{\tau+l}, \qquad \delta_\tau = R_\tau + \Gamma V(s_{\tau+1}) - V(s_\tau).$$

This construction is significant because it avoids treating ACT as if it were a step-wise policy under a conventional PPO wrapper. Instead, PAC-ACT changes the MDP granularity so that the optimization unit matches the policy’s chunked action generation.

3. ACT-transferred actor-critic architecture

PAC-ACT builds both actor and critic directly from the pretrained ACT network. The actor preserves the pretrained ACT backbone—visual encoder, Transformer encoder, and action decoder—but removes the CVAE latent module. The stated reason is that PPO already injects exploration through a Gaussian policy, so retaining the CVAE would introduce an additional source of stochasticity and create gradient conflict with the baseline constraint. The actor outputs the mean of a diagonal Gaussian: $46$0 with learnable diagonal covariance $46$1. The reported initialization of the log standard deviation differs by task, including $46$2 for Metal Touch and $46$3 for Square Assembly (Pang et al., 10 Jul 2026).

The critic reuses the pretrained ACT feature extractor—visual encoder and Transformer encoder—but removes the decoder, since value estimation does not require action generation. The resulting value function is

$46$4

The observation space is multimodal, combining front and wrist RGB images with 6-DoF joint-angle encoding. The paper argues that the transferred encoder already contains task-relevant structure, so preserving it while simplifying the critic improves stability. The ablation evidence reported later is consistent with that claim: the encoder-only critic outperforms a decoder-retaining alternative.

4. Hybrid behavior-prior constraint and training objective

PAC-ACT regularizes RL fine-tuning with what the paper terms a hybrid behavior-prior constraint. This has two components: a KL penalty between successive policies and a reward-level penalty that keeps the fine-tuned policy close to a frozen pretrained ACT baseline $46$5 (Pang et al., 10 Jul 2026).

The training loss includes a trust-region style KL term,

$46$6

with $46$7. This term constrains distributional drift between PPO updates, which is particularly relevant because action chunks are high-dimensional.

The second component modifies the rollout reward: $46$8 with $46$9. This penalty keeps exploration near the pretrained ACT action manifold, which the paper presents as essential in sparse-reward or high-dimensional chunk spaces where unconstrained exploration can drift into unsupported behaviors.

The PPO objective is written over chunk actions: $K=100$0 where the probability ratio is

$K=100$1

The full loss combines the negative PPO objective, the KL term, critic regression, and entropy regularization: $K=100$2

For the Metal Touch environment, the dense reward is

$K=100$3

The force penalty is tied to the total force magnitude,

$K=100$4

A plausible implication is that PAC-ACT’s distinctive contribution lies less in PPO itself than in how PPO is constrained so that online RL improves safety and robustness without erasing the pretrained ACT behavior.

5. Training pipeline and benchmark environments

The training pipeline begins with offline ACT pretraining on expert demonstrations in the LeRobot framework; the main model uses 800 expert trajectories. PAC-ACT then initializes the actor from the pretrained ACT, initializes the critic from the ACT encoder, freezes the baseline ACT policy for the reward penalty, and performs online chunk-level PPO fine-tuning (Pang et al., 10 Jul 2026).

The primary benchmark is Metal Touch, a MuJoCo precision-contact environment in which a 6-DoF arm with a probe-like end-effector must touch designated regions on a rubber-metal component. To model industrial uncertainty, the object is randomly shifted by up to $K=100$5 cm per episode. Three tasks are emphasized:

• Diamond: touching four side points in sequence.
• Cross: repeated center crossings with observation-action ambiguity.
• Contour: continuous motion along the inner boundary of the ring; this is described as the hardest and most contact-intensive task.

The policy observes front and wrist RGB images and 6 joint angles, while force/torque readings are available in simulation for training and evaluation. The control frequency is 10 Hz.

PAC-ACT is also evaluated on Square Assembly from robomimic as a transfer and scalability test beyond Metal Touch. Baseline comparisons include ACT, Diffusion Policy, and the vision-language-action baseline $K=100$60.5.

6. Reported performance, ablations, and limitations

The reported empirical results emphasize three properties simultaneously: task success, completion efficiency, and force safety (Pang et al., 10 Jul 2026). On the Contour task, success rates are reported as 60.0% for ACT, 60.0% for Diffusion Policy, 79.0% for $K=100$70.5, and 100.0% for PAC-ACT. On Square Assembly, the corresponding values are 51.2%, 77.8%, 62.6%, and 98.2%.

Completion efficiency improves as well. On Contour, average completion steps decrease from 485.4 to 170.6, and episode duration decreases from 48.5 s to 17.1 s after fine-tuning. The system also retains ACT’s deployment profile: mean inference latency 88.1 ms, P95 latency 125.6 ms, and GPU memory 2.30 GB. The cited comparison point for $K=100$80.5 is 44.15 GB peak GPU memory.

Force-safety measurements are particularly prominent. Relative to ACT on Contour, PAC-ACT reduces median peak force from 105.40 N to 20.74 N, mean force from $K=100$9 N to $a_t$0 N, and maximum peak force from 8452.5 N to 120.9 N. The proportion of force readings above $a_t$1 decreases from 4.6% to 0.1%, and the fraction of episodes with any reading above $a_t$2 decreases from 20% to 6%. The abstract summarizes the first of these effects as a 46 times reduction in the proportion of readings above the $a_t$3 threshold.

The ablation study attributes these gains to specific architectural decisions. A decoder-free critic attains $a_t$4 final-window mean success rate, compared with $a_t$5 for a critic that retains the Transformer decoder. An actor that removes the CVAE reaches $a_t$6, whereas the CVAE-retained actor reaches $a_t$7. In the sparse-reward setting, retaining the KL behavior-prior constraint yields 100% rollout success and preserves structured contour-following behavior under randomized initial poses; removing that constraint leads to 22.0% cumulative total success and a final rollout window of 72.7%. This suggests that success-rate statistics alone can obscure degradation in trajectory structure when the behavior prior is absent.

The reported limitations are also explicit. The evaluation is mostly simulation-based; sim-to-real transfer is not yet established. The perturbations are mainly randomized object positions rather than broader changes such as lighting, viewpoint, or dynamics. The strongest results use dense rewards, although sparse-reward ablations are included. The removal of the CVAE may simplify optimization at the cost of reduced multimodality, and the authors present the method as oriented toward industrial precision-contact control rather than as a universal policy-learning framework.

In that sense, PAC-ACT is best understood as a deployment-focused post-training method: it preserves the low-latency, low-memory, temporally continuous structure of Action Chunking Transformers, while using chunk-level actor-critic optimization and explicit behavior priors to improve robustness and force-constrained safety in contact-rich industrial manipulation.