Optimal allocation of parallel resources between batch size and sequence-length parallelization in MCMC

Ascertain when, given fixed accelerator resources, it is more beneficial to run more independent Markov chain Monte Carlo chains in parallel versus to allocate those resources to parallelizing across the sequence length using Newton-based methods (DEER or quasi-DEER) to run longer chains, and characterize the regimes and criteria that determine this allocation trade-off.

Background

The proposed approach parallelizes MCMC across the sequence length using Newton-based methods, trading increased memory and work (from many parallel function and gradient evaluations) for decreased time complexity. This introduces a new resource allocation choice: using hardware parallelism to batch more independent chains or to accelerate individual chains by parallelizing across time. The authors note that for targets with high memory evaluation costs, the parallel-in-time approach may be disadvantageous, underscoring the need for principled guidance on resource allocation.