Pretrain-Adaptive Time-step Adjustment (PATA)

• The paper introduces PATA, a dynamic strategy that learns scene-specific inference time steps in spike-based NeRFs to balance rendering quality and computational efficiency.
• It replaces fixed time steps with a learnable parameter optimized through a two-phase training process, significantly cutting computation and energy consumption.
• Key innovations, including hybrid temporal weighting and adversarial fine-tuning, achieve reductions of up to 64% in steps and 61.55% in power usage while preserving near-ANN fidelity.

Pretrain-Adaptive Time-step Adjustment (PATA) is a dynamic time step training strategy for spike-based Neural Radiance Fields that treats the inference time step as a learnable, scene-specific quantity rather than a fixed hyperparameter. Introduced in a spike-based adaptation of Instant-NGP, PATA is designed to preserve rendering fidelity while reducing the temporal depth required by Spiking Neural Networks (SNNs) during inference, thereby lowering spike-based computation and power. The reported outcome is scene-adaptive inference with variable time steps, with reductions of inference time steps by up to 64\% and running power by 61.55\% while preserving near-ANN rendering fidelity (Lin et al., 30 Jul 2025).

1. Problem formulation and motivation

PATA is motivated by two observations. First, NeRF and Instant-NGP remain compute intensive because they rely on dense ray sampling and many floating-point operations. Second, SNNs are energy-efficient, but their performance depends strongly on the number of discrete time steps: more time steps generally improve temporal information capacity and rendering quality, but they also increase computation and energy. In spike-based NeRF, the paper reports a strong correlation between rendering quality and the maximum time step $T$, but also emphasizes that this relation is scene-dependent: simple scenes may reach good PSNR with very few time steps, whereas complex scenes need more (Lin et al., 30 Jul 2025).

Within this formulation, PATA aims to learn a scene-specific target time step that balances fidelity and efficiency. The central claim is not merely that fewer time steps are desirable, but that a fixed $T$ is suboptimal across scenes. The method therefore makes the inference time step itself an optimizable quantity.

This design distinguishes PATA from fixed-step spike-based NeRF training. A fixed-step protocol commits to one temporal budget for all scenes, whereas PATA learns a per-scene operating point on the quality–efficiency curve. The paper further notes that this is done within a unified GPU-friendly pipeline: all points in a scene use the same chosen time step, which preserves parallelism and differs from sample-wise early stopping methods that can hurt batch throughput.

2. Backbone architecture and spiking formulation

PATA is anchored to the Instant-NGP architecture rather than introducing a new NeRF backbone. The grid-based NeRF formulation maps 3D coordinates into hashed multiresolution grid features and then processes them with a density network and a color network: $$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$ For ray rendering, the method uses standard volume rendering: $$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$

$$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$

The architectural change is that the ANN MLPs are replaced with PLIF-based spiking layers (Lin et al., 30 Jul 2025).

The membrane update is written as

$$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$

with

$i(t)=W^l s^{l-1}(t)$

Spikes are generated by thresholding,

$s^l(t)=H(v^l(t)-\theta^l)$

followed by the soft reset

$v^l(t)=v^l(t)-s^l(t)\theta^l$

and the non-differentiability of $H(\cdot)$ is handled using the surrogate gradient

$T$0

with $T$1.

The paper states that PATA retains the original input and output dimensions of Instant-NGP and adds only a small number of trainable parameters, specifically shared decay factors $T$2 across PLIF layers, with only 5 extra trainable parameters reported. It also inherits Instant-NGP’s occupancy-grid sampling, which reduces the sampled points and therefore the memory burden over time. A related systems mechanism, Max Time-Step Scaling, starts with a smaller $T$3 for a few epochs and restores $T$4 once sampling stabilizes; this is reported to reduce peak memory by up to 69\%.

3. Learnable time-step adaptation and hybrid temporal weighting

A key technical issue identified by the paper is that ordinary LIF or PLIF temporal weighting makes later time steps dominate the final output. The output at time step $T$5 is analyzed as

$T$6

where $T$7 is exponential for density and sigmoid for color. Under standard modes, the time-step weights are

$T$8

and these weights still increase toward later steps. The stated consequence is that truncating time steps hurts quality because the last steps carry too much influence (Lin et al., 30 Jul 2025).

To reduce this dependence on late temporal contributions, PATA introduces a hybrid input mode: $T$9 This yields

$$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$0

and the paper notes that these weights sum to $$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$1 for $$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$2. The practical effect is that the initial time step receives a larger relative contribution while later steps are suppressed, which makes shorter inference windows less damaging.

The learnable temporal budget is represented by a trainable target time step $$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$3, separate from the fixed maximum time step $$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$4. Since inference uses integer steps, the continuous parameter is quantized as

$$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$5

This makes $$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$6 a latent optimization target during training and $$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$7 the executable scene-specific step count used at inference.

4. Two-phase training and objective design

PATA is explicitly described as a two-phase training strategy. In the pretraining phase, the model is trained with the full maximum time step $$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$8. In the adaptation phase, both the spike-based model and the target time step $$\mathbf{f} = \mathrm{interp}(\mathbf{x}, \Phi_{\theta}), \quad h, \sigma = MLP_{density}(\mathbf{f}), \quad \mathbf{c} = MLP_{color}(\mathbf{d}, h)$$9 are optimized (Lin et al., 30 Jul 2025).

The stated rationale for the two phases is that direct optimization of $$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$0 from scratch is unstable for two reasons: early in training, the model is not yet expressive enough to judge how many steps are needed, and $$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$1 can collapse prematurely to a small, suboptimal value. The pretraining stage therefore builds a strong radiance-field representation first, after which time-step adaptation becomes stable.

In the first stage, the rendering loss at $$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$2 contains three named components: $$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$3, $$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$4, and $$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$5. The data specifies

$$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$6

and gives the consistency term and Cauchy regularizer, with $$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$7, $$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$8, and $$\alpha_i = 1 - \exp(-\sigma_i \delta_i), \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j)$$9. The Cauchy regularizer is described as encouraging sparse density predictions.

In the second stage, the objective is

$$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$0

The rendering term is evaluated at the adapted step $$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$1. Because $$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$2 is discrete and can change abruptly, the paper introduces a smoothed target-step objective by interpolating between neighboring integer steps: $$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$3 where $$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$4. This smooths the loss landscape near integer transitions.

The distillation term is

$$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$5

with $$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$6 acting as a teacher signal for the lower-step output. The time-step control itself is governed by the adversarial term

$$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$7

with $$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$8 in experiments. The “extra” term is written as

$$\hat C(\mathbf{r}) = \sum_{i=1}^{N} T_i \alpha_i \mathbf{c}_i$$9

and the penalty term as

$$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$0

The paper states that $$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$1 directly controls the operating point on the quality–efficiency curve.

5. Training protocol, inference behavior, and reported performance

The experimental setup uses an Instant-NGP-based spike NeRF implemented in PyTorch + SpikingJelly on an NVIDIA 4090 GPU. The maximum time step is $$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$2, the initial target time step after pretraining is $$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$3, the schedule is 10,000 iterations for stage 1 and 30,000 iterations for stage 2, the maximum step at the start of each stage is 2 for the first epoch, then 8 afterward, the PLIF decay factor is $$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$4, and the spike threshold is $$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$5. Evaluation is reported on Synthetic-NeRF with 8 synthetic scenes at $$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$6 resolution and on Mip-NeRF 360, which contains real-world scenes with higher geometric and textural complexity (Lin et al., 30 Jul 2025).

After training, each scene is associated with a learned target step $$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$7. Inference then uses this scene-specific step count rather than a universal fixed $$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$8. The paper summarizes the resulting behavior as follows: simple scenes converge to smaller $$v^l(t) = \begin{cases} (1-\frac{1}{\tau})v^l(t-1) + \frac{1}{\tau} i(t), & \text{decay input} \ (1-\frac{1}{\tau})v^l(t-1) + i(t), & \text{non-decay input} \end{cases}$$9, reducing computation and power, while complex scenes retain larger values to preserve detail and PSNR.

The main quantitative trade-off is reported by varying $i(t)=W^l s^{l-1}(t)$0:

Setting	Synthetic-NeRF	Mip-NeRF 360
INGP-NeRF	PSNR 32.14, SSIM 0.959, energy 1910.54 mJ	PSNR 25.48, SSIM 0.668, energy $i(t)=W^l s^{l-1}(t)$1 mJ
PATA, $i(t)=W^l s^{l-1}(t)$2	PSNR 32.45, SSIM 0.959, time step 5.125, energy 1217.99 mJ	PSNR 25.30, SSIM 0.667, time step 6.0, energy $i(t)=W^l s^{l-1}(t)$3 mJ
PATA, $i(t)=W^l s^{l-1}(t)$4	PSNR 32.21, SSIM 0.957, time step 3.625, energy 887.20 mJ	PSNR 25.09, SSIM 0.646, time step 4.857, energy $i(t)=W^l s^{l-1}(t)$5 mJ
PATA, $i(t)=W^l s^{l-1}(t)$6	PSNR 32.04, SSIM 0.956, time step 2.875, energy 779.12 mJ	PSNR 25.01, SSIM 0.641, time step 4.0, energy $i(t)=W^l s^{l-1}(t)$7 mJ

The paper also compares PATA with spike-based baselines on Synthetic-NeRF: Spiking-NeRF at 30.41 PSNR with 256 time steps, Mixed Spiking-NeRF at 31.61 PSNR with 4 steps, SpikingNeRF-D at 31.64 PSNR with 4 steps, SpikingNeRF-T at 32.45 PSNR with 1 step, and PATA at 32.21 PSNR with 3.625 steps. Against ANN methods on Synthetic-NeRF, PATA reaches 32.21 PSNR and 0.957 SSIM; on Mip-NeRF 360 it reaches 25.09 PSNR and 0.646 SSIM while retaining the spike-based energy advantage.

Energy is computed from both spike-based operations and FLOPs: $i(t)=W^l s^{l-1}(t)$8

$i(t)=W^l s^{l-1}(t)$9

$s^l(t)=H(v^l(t)-\theta^l)$0

$s^l(t)=H(v^l(t)-\theta^l)$1

Within this accounting, the reduction in time-step budget directly lowers SOPs and total energy.

6. Ablations, interpretation, and relation to broader adaptive time-step work

The ablations clarify which components are functionally necessary. For the hybrid input mode, the paper reports that decay-only performs worst because the $s^l(t)=H(v^l(t)-\theta^l)$2 scaling weakens accumulation, non-decay-only can overflow or become unstable, and hybrid provides the best trade-off by improving early-step contribution (Lin et al., 30 Jul 2025).

For two-stage training, the ablation gives baseline full-$s^l(t)=H(v^l(t)-\theta^l)$3 training at 32.12 PSNR, 0.957 SSIM, 8 steps; without pretraining at 31.59 PSNR, 0.954 SSIM, 3.625 steps; and full PATA at 32.21 PSNR, 0.957 SSIM, 3.625 steps. A common misunderstanding would be that direct adaptation alone is sufficient; the reported result indicates that direct adaptation can reduce steps but loses about 0.53 dB PSNR, so the pretraining phase is central to preserving fidelity.

For the adversarial loss, the ablation reports without $s^l(t)=H(v^l(t)-\theta^l)$4: 31.69 PSNR, 1.625 steps; without $s^l(t)=H(v^l(t)-\theta^l)$5: 32.61 PSNR, 6.625 steps; without $s^l(t)=H(v^l(t)-\theta^l)$6: 32.02 PSNR, 5.875 steps; and the full method: 32.21 PSNR, 3.625 steps. The stated interpretation is that $s^l(t)=H(v^l(t)-\theta^l)$7 stabilizes quality when reducing steps, $s^l(t)=H(v^l(t)-\theta^l)$8 pushes the model toward shorter inference, and together they produce the best balance.

PATA also belongs to a broader class of methods that elevate temporal discretization from a static setting to an adaptive control variable. A plausible conceptual analogy is the two-stage receding structure in humanoid gait generation, where nominal step timing is selected first and then adapted online to disturbances; the paper on step timing adjustment explicitly argues that step location adjustment alone is not enough and that step timing should also be adapted (Khadiv et al., 2016). Another plausible parallel appears in adaptive time step control for multirate infinitesimal methods, where the controller adjusts both the macro time step $s^l(t)=H(v^l(t)-\theta^l)$9 and the inner multirate ratio $v^l(t)=v^l(t)-s^l(t)\theta^l$0 to balance separate slow- and fast-scale error sources (Fish et al., 2022). These works address different domains, but they suggest a common pattern: temporal allocation is treated as a first-class optimization variable rather than an implementation constant.

In that broader sense, PATA’s distinctive contribution is domain-specific. It does not propose a new radiance-field parameterization, nor does it replace Instant-NGP’s sampling and rendering structure. Instead, it learns how much spiking temporal depth a given scene actually requires, using hybrid temporal weighting, a pretrain-then-adapt schedule, distillation from the full-step output, and an explicit quality–efficiency control term. The result is a scene-adaptive spike-based NeRF in which time-step selection becomes part of the learned representation rather than a manually fixed deployment choice.