FLOP‑optimal selection of μ_rec and μ_bwd for looped training

Determine the FLOP‑optimal choices of μ_rec (the mean number of recurrent forward steps during pretraining) and μ_bwd (the mean number of recurrent backpropagation steps during pretraining) for looped models such as Parcae under fixed compute and parameter budgets, identifying how these hyperparameters should be set to optimize training efficiency and generalization.

Background

The authors empirically study how varying μ_rec and μ_bwd affects performance, observing that increasing μ_rec without sufficiently increasing μ_bwd can hurt validation loss at both low and high test-time recurrences. They adopt a heuristic μ_bwd = ⌈μ_rec / 2⌉ based on preliminary experiments.

They explicitly state that identifying the FLOP‑optimal settings of these training hyperparameters remains to be addressed in future work, indicating a concrete open optimization problem tied to training efficiency and effectiveness.