Localized Empirical Entropy

• Localized empirical entropy is defined as evaluating entropy within local, sample-specific or spatial neighborhoods, enabling adaptive quantification of statistical complexity.
• It underpins methodologies across nonparametric statistics, dynamical systems, and thermodynamics by linking local entropy measures to minimax risk, multifractal analysis, and entropy production.
• Applications include local regularization in deep learning, empirical risk minimization, and intrinsic dimension estimation, offering actionable insights in high-dimensional and heterogeneous systems.

Localized empirical entropy is a broad, methodologically rich concept that encompasses both the spatial or sample-dependent measurement of entropy in probability measures, statistical models, dynamical systems, and deep learning representations. It underpins the quantitative analysis of statistical complexity, information content, and resource scaling across varied domains such as nonparametric statistics, dynamical systems, nonequilibrium thermodynamics, and representation learning. Formal constructions include empirical covering entropies on sample paths, local entropy averages for geometric or multifractal analysis, data-driven localization of entropy production in stochastic trajectories, and intrinsic dimensional entropy adapted to continuous high-dimensional data.

1. Localized Empirical Entropy: General Notions and Variants

Localized empirical entropy refers to entropy-type quantities evaluated or estimated within a spatial, functional, or sample-dependent neighborhood, rather than globally across a support, function class, or underlying probabilistic ensemble. It appears in distinct forms:

• Metric entropy localized by samples or covariate neighborhoods, used in empirical process theory and nonparametric statistics to capture the metric covering structure relevant to the realized data rather than the entire underlying space (Bilodeau et al., 2021).
• Local entropy averages on dyadic or metric balls, prominent in fractal geometry and dimension theory to relate empirical entropy measurements to local geometric dimensions and heterogeneity (Sahlsten et al., 2011).
• Localized entropy production rates in nonequilibrium stochastic processes, quantifying entropy production within specific regions of phase space or along time-resolved segments of a trajectory (Das et al., 26 Mar 2025).
• Local intrinsic dimensional entropy (ID-Entropy), measuring the average dimension of the effective data manifold in a neighborhood around each sample, employed for robust entropy estimation in continuous spaces (Ghosh et al., 2023).
• Local entropy regularization in deep learning loss functions, where entropy is used as a data-dependent, parameter-localized smoothing or flatness penalty for optimization (Trillos et al., 2019).

These constructions generally facilitate improved adaptivity, sample-efficiency, or physical interpretability compared to global entropy measures, especially in high-dimensional, heterogeneous, or complex systems.

2. Minimax Rates and Localized Empirical Entropy in Statistics

In statistical learning theory, localized empirical entropy plays a decisive role in characterizing minimax risk rates for estimation, particularly for complex, high-dimensional, or nonparametric function classes under sample-dependent design:

• For conditional density estimation, classic uniform metric entropy is typically infinite or suboptimal when the covariate space is large or unbounded. Empirical localization remedies this by assessing Hellinger entropy only on the observed sample $\{x_{1:n}\}$, leading to the empirical Hellinger entropy:

$$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$

where $N_H$ denotes the covering number in the empirical Hellinger metric $d_{P_n}$ (Bilodeau et al., 2021).

• The critical complexity parameter is the localized entropy of $\epsilon$-neighborhoods around the true density $f_0$, yielding fixed-point equations of the form $H_{P_n}(\hat r, \mathcal{F}) \asymp n \hat r^2$, which directly determine the minimax KL-risk rate $R_n \asymp \hat r^2$.
• Empirical entropy localization permits matching upper and lower risk bounds for function classes with polynomial or logarithmic entropy growth, and critically, avoids the "curse of dimensionality" present in worst-case (uniform) entropy calculations.
• The methodology connects to local Rademacher complexity, which provides sharp uniform concentration inequalities essential for controlling the empirical-to-expected risk transitions in high-complexity model classes.

In summary, localized empirical entropy provides a sample-adaptive complexity measure that is both statistically minimax-optimal and computationally feasible in high-dimensional or nonparametric settings (Bilodeau et al., 2021).

3. Local Entropy Averages and Dimensional Analysis of Measures

In geometric measure theory and multifractal analysis, local entropy averages serve as a fundamental tool for connecting statistical entropy averages to local (pointwise) dimensions:

• For a Borel measure $\mu$ on $[0,1)^d$, define the dyadic cube decomposition $$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$0 and the entropy over refinements:

$$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$1

where $$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$2 is the Shannon entropy function (Sahlsten et al., 2011).

• The local entropy average at $$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$3 is then

$$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$4

with the result that, for $$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$5-almost every $$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$6,

$$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$7

and similarly for the lower local dimension.

• This entropy-averaging mechanism unifies proofs and quantitative bounds for local homogeneity, conical density, and mean porosity, replacing ad-hoc covering or combinatorial arguments by entropy-based dimension estimation techniques (Sahlsten et al., 2011).
• Empirical implementations involve plugging in sample-based frequencies to estimate local entropies at various scales, with robustness to grid choice and reliability in high dimensions.

The local entropy average is thus central to both theoretical and empirical quantification of multifractal structure and geometric heterogeneity of measures.

4. Local Shannon Entropy and Nonequilibrium Thermodynamics

The localization of entropy at the level of microstates or trajectories is foundational in modern statistical physics and nonequilibrium thermodynamics:

• Rather than ensemble-level (global) entropy, local Shannon entropy $$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$8 is defined for the probability $$H_{P_n}(\epsilon, \mathcal{F}) = \sup_{x_{1:n}} \log N_H(\epsilon, \mathcal{F}, x_{1:n}),$$9 of a microstate $N_H$0 at time $N_H$1 within a local sub-ensemble of histories ending at $N_H$2 (Jinwoo et al., 2014).
• The local information $N_H$3, counting the total weight of trajectories ending at $N_H$4, delivers a trajectory-independent but space–time-specific potential.
• The local free energy combines energy and local entropy as $N_H$5, with implications for the generalization of the Boltzmann-Gibbs formula and Landauer's principle at the microstate level.
• Local Crooks-Jarzynski-type relations and exact dynamical equations in terms of information flow underscore the status of localized entropy as a true state-function governing non-equilibrium relaxation.

This theoretical framework justifies the meaningful definition of entropy and related thermodynamic potentials at any particular location in phase space and time, supporting empirical and simulational studies of small-scale and fluctuating systems (Jinwoo et al., 2014).

5. Data-Driven Localization of Entropy Production in Nonequilibrium Systems

Localized entropy production provides a quantitative, empirically inferable mapping from stochastic dynamics to spatially and temporally resolved dissipative structure:

• For Langevin-type systems, the local entropy production rate density is

$N_H$6

where $N_H$7 is the local probability current and $N_H$8 is the local thermodynamic force (Das et al., 26 Mar 2025).

• Recent methodology combines the thermodynamic uncertainty relation (TUR) in the vanishing time-scale limit with neural network representations to variationally infer $N_H$9 from trajectory data.
• The resulting spatial and temporal resolution allows mapping entropy production landscapes in both stationary and time-varying, high-dimensional, and strongly non-equilibrium systems.
• Empirical demonstrations include harmonic oscillators, active hair-cell bundle models, and bit-erasure experiments, illustrating the utility of localized entropy production as an empirical tool.

The fusion of statistical physics principles with machine learning architectures yields a general-purpose approach for reconstructing dissipation patterns at the finest operational scale (Das et al., 26 Mar 2025).

6. Intrinsic Dimensional Entropy: Structure-Preserving Local Entropy for Continuous Spaces

The inadequacy of statistic-based global entropy on $d_{P_n}$0 motivates dimension-sensitive, locally defined entropy constructs:

• ID-Entropy (Intrinsic Dimensional Entropy) is defined as

$d_{P_n}$1

where $d_{P_n}$2 is the local intrinsic dimension at $d_{P_n}$3, estimated via, e.g., the Levina–Bickel $d_{P_n}$4-nearest neighbor method (Ghosh et al., 2023).

• Unlike classical entropy, which is sensitive to the measure's spread over the entire space, ID-Entropy depends on the local manifold structure and is bounded above by true data dimension.
• Properties include invariance under homeomorphism, subadditivity, a deterministic data-processing inequality, and stability under coordinate changes. Mutual and conditional versions extend the theory, paralleling Shannon-type information.
• In machine learning, ID-Entropy of hidden layers tightly controls generalization bounds for both classifiers and auto-encoders (e.g., lower $d_{P_n}$5 implies reduced generalization gap under Lipschitz conditions), and is empirically predictive of test error across architectures and data regimes.
• The measure is robust to noise, monotone under transformations, and effectively characterizes causal structure in generative models.

Algorithmic estimation, based on efficient $d_{P_n}$6-NN averaging, makes ID-Entropy a scalable, practical metric for continuous and high-dimensional data (Ghosh et al., 2023).

7. Local Entropy in Deep Learning: Variational and Algorithmic Approaches

In deep learning, localized entropy functionals are incorporated both as loss regularizers and as diagnostic tools for flatness and generalization:

• Local-entropy regularized loss for a parameter vector $d_{P_n}$7 and unregularized loss $d_{P_n}$8 is given by

$d_{P_n}$9

where $\epsilon$0 is a Gaussian kernel centered at $\epsilon$1. This locally averages the loss landscape around each point (Trillos et al., 2019).

• Optimization proceeds by alternating between:
  1. Forming the Gibbs posterior $\epsilon$2;
  2. Performing a moment-matching update by setting $\epsilon$3.

• This iterative shift + fit scheme replaces explicit gradient computation with sampling-based estimation (via importance sampling, SGLD, or MCMC), yielding a gradient-free, parallelizable alternative to classical backpropagation.

• The method generalizes to heat-regularized losses, which differ in the variational character and functional monotonicity properties.

Localized empirical entropy thus underlies both regularization and interpretability strategies within high-capacity neural networks, with theoretical implications for convergence and flatness (Trillos et al., 2019).

---

Localized empirical entropy, in its statistical, geometric, thermodynamic, and machine learning incarnations, provides a rigorous and operationally significant measurement of complexity, unpredictability, and informational content tailored to local, sample-dependent, or physically resolved scales. Its empirical estimability, connection to nonasymptotic risk, and adaptivity to data structure establish it as a foundational instrument across mathematical, statistical, and applied disciplines.