Local Action Variance: Metrics & Applications

• Local action variance is a context-specific metric that measures variability across actions, spatial regions, or data subsets.
• It integrates methodologies from RL, deep networks, statistical clustering, and cosmological inference to compute segment-wise uncertainties.
• Applications include enhanced exploration-exploitation in RL and improved boundary uncertainty in temporal action localization for better model interpretability.

Local action variance quantifies, in a local or context-specific sense, the dispersion or sensitivity of a quantity—such as expected return, predicted boundary location, or a cosmological parameter—with respect to either action choice, spatial region, or data subset. This concept appears across reinforcement learning (RL), deep learning for temporal action localization, statistics, and cosmological inference. Rigorous definitions, methodologies for computation, and significance in diverse domains are provided in peer-reviewed research (Karino et al., 2020, Xie et al., 2020, Solomon et al., 2020, Jr, 2015, Yue et al., 2020).

1. Formal Definitions Across Domains

Several formalizations of local action variance have been proposed, each tailored to the domain:

• Reinforcement Learning (RL): The local action variance at a state $s$ is formalized as the action-based variance of the state’s action-value function:

$$\mathrm{SI}(s) = \mathrm{Var}_{a\sim U(A)}\left[Q^\pi(s,a)\right] = \frac{1}{|A|}\sum_{a\in A} \left(Q^\pi(s,a) - \bar Q(s)\right)^2$$

where $\bar Q(s) = \frac{1}{|A|}\sum_{a} Q^\pi(s,a)$ and $U(A)$ is the uniform distribution over actions (Karino et al., 2020).

• Variance Propagation in Deep Networks: In variance-aware networks (VANs) for temporal action localization, "local" action variance refers to the per-segment feature variance at different network layers and is interpreted as the ambiguity of action boundaries within video segments. Each feature vector $x_j$ is modeled as $x_j \sim \mathcal{N}(\mu_j,\sigma_j^2)$ with propagation of $(\mu, \sigma^2)$ analytically through the network (Xie et al., 2020).
• Clustered/Distributional Statistics: In $k$-variance, a "local" version of variance, the quantity

$$\mathrm{Var}_k(\mu) = \frac{1}{2} \rho(k,d)\, \mathbb{E}_{X,Y\sim\mu^k} \left[ W_2^2(\mu_k, \nu_k) \right]$$

is defined by matching batches of $k$ samples from distribution $\mu$; as $k$ increases, variance reflects local rather than global structure (Solomon et al., 2020).

• Cosmological Parameter Inference ("Hubble Variance"): Local variance can refer to the spatial directionality of a cosmological parameter (e.g., $H_0$) estimated in different sky hemispheres, with maximal hemispherical variance $$\delta H_0 = H_0^{\mathrm{max}} - H_0^{\mathrm{min}}$$ (Jr, 2015).

A synthesis is that local action variance is always a context-dependent dispersion metric that conditions either on specific states, spatial locations, input segments, or restricted clusters.

2. Methodologies for Calculation and Propagation

The calculation of local action variance depends on the application:

• RL (“State Importance”): Computed directly from the spread of Q-values across available actions for the current policy, with the uniform prior on actions in the simplest case (Karino et al., 2020). For continuous actions, this may involve Monte Carlo sampling.
• VANs (Deep Learning): Local action variance is estimated via variance-aware pooling on input segments, yielding $(\mu, \sigma^2)$ for each segment, which are propagated analytically through layers (fully connected, normalization, ReLU, etc.) using specific propagation equations. Output variances serve both as a regression target (with KL divergence loss) and as a measure of uncertainty (Xie et al., 2020).
• $k$-Variance: Draws two independent batches of $k$ i.i.d. samples from a distribution; solves a bipartite matching linear assignment problem to calculate the optimal transport cost (matching cost), then averages across trials, rescaling by appropriate dimension-dependent factors (Solomon et al., 2020).
• Cosmology ("Hemispherical Analysis"): Pixelizes the sky, fits the parameter (e.g., $H_0$) in each hemisphere via $\chi^2$ regression, and computes the maximal range across all directions (Jr, 2015).
• Variance Control in Policy Gradients: Local action variance refers to the variance of the on-policy gradient estimator's action-value component for each state-action pair. Analytical and empirical variance reductions are achieved using correlated pseudo-actions, zero-mean critic combinations, and sparsification (Yue et al., 2020).

3. Significance in Interpretability and Exploration

Local action variance is instrumental in several key tasks:

• Identifying Critical Points in RL: High action-variance states are "critical states," where action choice has a large impact on expected return—these states are both bottlenecks for learning and key for interpretability. Accelerated RL is achieved by exploiting aggressively at high-variance states and exploring elsewhere, yielding significant speedups in tabular and deep RL (Karino et al., 2020).
• Expressing Uncertainty in Action Localization: In temporal action localization, pooling variances reveal inherently ambiguous (high-variance) regions of a video; propagating these through networks enables the model to express its uncertainty quantitatively and avoid over-penalization for ambiguous boundaries (Xie et al., 2020).
• Statistical Locality in Distributional Analysis: $k$-variance captures local spread, multi-modality, and clustering properties of data distributions, effectively differentiating between local and global dispersion and facilitating distribution summary at multiple scales (Solomon et al., 2020).

4. Algorithms and Quantitative Impacts

Implementations utilizing local action variance achieve concrete improvements:

• RL Critical-State Exploitation Rule: The algorithm ranks states by SI(s), labels the top-$q$ fraction as critical, and applies modified $\varepsilon$-greedy with increased exploitation probability $k$ at those states. This reduces learning steps by 70%+ in discrete environments and provides statistically robust learning benefits in Atari and MuJoCo tasks (Karino et al., 2020).
• CARSM-PG Gradient Estimation: Constructs correlated pseudo-actions via Dirichlet draws; combines their Q-values to exploit negative covariance for variance cancellation and sparsifies gradient updates where local signal vanishes. The resulting gradient variance is empirically 3–10$\times$ lower than baseline estimators, with a theoretically demonstrated reduction by $(1-2/C)$ per-dimension for large action alphabet size $C$ (Yue et al., 2020).
• Variance-Aware Networks: The propagation-based VAN achieves higher mAP in action localization benchmarks with no parameter or FLOP overhead and outperforms simply concatenating variance features or predicting output variances alone. Variance propagation specifically down-weights regression loss where boundary uncertainty is high, focusing supervised learning on reliably scored events (Xie et al., 2020).
• $k$-Variance Empirical Computation: Sampling-based evaluation of Var$_k$ is feasible for moderate $k,d$; complexity is $O(k^2 d + k^3)$ per trial, and McDiarmid's inequality quantifies estimator accuracy (Solomon et al., 2020).

5. Theoretical and Empirical Properties

Local action variance metrics satisfy key invariance and convergence properties:

Domain	Local Action Variance Formula	Key Properties
RL	SI(s) = Var$_a$ Q$^\pi$(s,a)	Peaks at critical states; interpretable; accelerates exploration-exploitation (Karino et al., 2020)
VANs	Segment-wise input/output $\sigma^2$	Expresses boundary ambiguity; loss-attenuation; supports uncertainty-aware inference (Xie et al., 2020)
$k$-variance	Var$_k(\mu)$ from $k\times k$ batch matching	Shift/scale equivariance; recovers variance for $k=1$; cluster-aware for larger $k$ (Solomon et al., 2020)
Cosmology	$$\delta H_0 = H_0^{\mathrm{max}} - H_0^{\mathrm{min}}$$	Directional anisotropy; detects bulk flow impacts; moderate statistical significance (Jr, 2015)
Policy Gradients	Per-dimension estimator variance of Q-combined gradients	Unbiasedness; negative covariance for variance reduction; empirical minimization (Yue et al., 2020)

Convergence and monotonicity: For $k$-variance, Var$_1$ recovers classical variance, Var$_k$ increases with $k$ and plateaus as $k\rightarrow \infty$, revealing cluster/local structure. In critical states of RL, SI(s) becomes sharply localized near bottleneck or decision states. In variance-aware temporal action localization, high predicted $\hat\sigma$ aligns with visually or kinematically ambiguous boundaries.

6. Practical Considerations and Limitations

Applicability and efficiency trade-offs are domain-dependent:

• RL Critical-State Methods: Sliding window or batch-based SI(s) computation is efficient in tabular and deep RL. Selection of $q$ (critical set fraction) and $k$ (exploitation probability) directly affects performance and policy interpretability. High SI(s) may be rare in smooth reward landscapes, and continuous actions require sampling-based estimation.
• Variance-Aware Networks: Propagation introduces no new FLOPs or parameters (for propagation-based VAN), but approximations (e.g. for ReLU) may be sub-optimal if variance is not axis-aligned. Output variances can directly serve user-facing uncertainty annotations.
• $k$-Variance: Computational burden increases as $k^3$; empirically, moderate $k$ suffices for local structure detection. In clustering applications, sweeping $k$ reveals elbow points for scale selection.
• Cosmological Parameter Local Variance: Sky coverage, measurement uncertainty, and sample density limit statistical significance. Despite alignment with bulk flow, results remain at moderate (68–80%) confidence (Jr, 2015). Larger/more uniform sampling is necessary for robust inference.
• Policy Gradient Variance Control: The Dirichlet-share method is efficient for multi-dimensional, moderately sized action spaces. Sparsification further prunes uninformative gradient dimensions, but is less effective for small $C$.

7. Broader Implications and Interpretability

Local action variance acts as a universal sensitivity metric, detecting and explaining regions of large impact for both algorithms and scientific models. In RL, it isolates decision states where agent intervention is critical, underpinning candid model explanations. In temporal action detection, it quantifies epistemic uncertainty, assisting model deployment and reliability. $k$-variance provides a theoretically robust, cluster- and locality-sensitive generalization of classical variance for statistical shape summarization. In cosmology, local variance-based analyses can reveal spatial anisotropies and physical phenomena that global averaging would conceal.

Taken together, these developments underscore local action variance as a central concept in model interpretability, exploration strategy, variance reduction methodology, and spatially-resolved scientific inference (Karino et al., 2020, Xie et al., 2020, Solomon et al., 2020, Jr, 2015, Yue et al., 2020).