The Okinawa Lectures on Entropy

Abstract: After a historical introduction, the most important classical and quantum entropies are introduced as constructions in classical and quantum probability theory. Classical entropies are studied from large deviation theory, including theorems of Sanov, Cramér, Gärtner-Ellis, and Varadhan, and are illustrated in some applications to both Boltzmannian and Gibbsian statistical physics. Quantum entropies superficially connect to classical entropies at the formula level, but more deeply do so via the crucial role of entropy in statistical hypothesis testing. The classical (relative) Kullback-Leibler entropy, its quantum counterpart introduced by Umegaki, as well as their deformations proposed by Renyi all fit naturally in this context. Quantum entropy faces the new problem of defining and computing the relative entropy of a pair of states on a subsystem, here formalized as a von Neumann algebra. This requires modular (aka Tomita-Takesaki) theory, which provides the framework for the relative quantum entropies introduced by Araki and Uhlmann (these encompass both the Kullback-Leibler and Umegaki entropies as special cases). To (re)define and compute these entropies in terms of density operators and traces, further constructions are needed, namely Haagerup's noncommutative L^p spaces. Von Neumann algebras also provide the setting for the Connes-Stormer-Narnhofer-Thirring entropy, which is a quantum version of the Kolmogorov-Sinai entropy in dynamical systems and ergodic theory, to which we also provide an introduction. This course was originally inspired by, and should be relevant to, black hole thermodynamics, although we discuss neither this application nor the second law. The course tries to be both mathematically rigorous and interesting to theoretical physicists. Prerequisites are undergraduate probability theory, functional analysis, and quantum theory.

• The paper presents an expository framework that treats relative entropy as a rate function governing fluctuations, distinguishability, coding performance, and dynamical complexity across classical and quantum systems.
• Classical results including Sanov’s theorem, Cramér’s theorem, the asymptotic equipartition property, and hypothesis-testing bounds show that Kullback–Leibler divergence determines exponential probabilities, coding thresholds, and error exponents.
• Quantum entropy is extended to infinite systems through modular theory, Araki relative entropy, and Haagerup L¹ spaces, while the lectures distinguish quantum Stein-type results from a still-unresolved general noncommutative analogue of Sanov’s theorem.

The paper presents an extensive set of graduate lecture notes that unify classical, quantum, and dynamical notions of entropy through probability theory, large deviations, hypothesis testing, and von Neumann algebras. Its central methodological claim is that entropy is most coherently understood not as an isolated scalar attached to a state, but as a rate function governing fluctuations, distinguishability, coding performance, or dynamical complexity. The notes are explicitly expository: the author states that they contain no substantially original results, with the principal contribution lying in the organization of the material and in the systematic juxtaposition of classical and quantum constructions (2608.14523).

Scope and organizing perspective

The course begins from the historical development of entropy in thermodynamics and statistical mechanics, proceeds through Shannon and Rényi information measures, and then develops the modern theory of relative entropies. The classical and quantum theories are not treated as parallel catalogues. Instead, the text identifies large-deviation theory as the principal organizing framework on the classical side and modular theory as the corresponding structural framework for general quantum systems.

This leads to a deliberate reversal of the usual pedagogical hierarchy. Shannon entropy and von Neumann entropy are presented as special cases of relative entropy, obtained when the reference state is uniform. The more fundamental objects are therefore the classical Kullback–Leibler divergence and its quantum counterparts, especially Umegaki relative entropy, Araki relative entropy, and Rényi-type deformations. Their significance is established operationally through hypothesis testing rather than solely through formal analogy.

The notes also distinguish sharply between several concepts often conflated under the word “entropy.” Classical statistical entropies are associated with fluctuating quantities and their atypical deviations from equilibrium, whereas phenomenological thermodynamic entropy is treated as an equilibrium concept. The paper consequently does not attempt to derive or formulate the second law of thermodynamics. It also excludes the entropy methods of partial differential equations and does not discuss black-hole thermodynamics, despite identifying black-hole physics as one of the motivations for the operator-algebraic material.

Historical development from thermodynamics to information theory

The historical introduction traces entropy from Clausius’s state function through Boltzmann’s kinetic and combinatorial approaches, Gibbs’s ensemble formalism, Einstein’s fluctuation analysis, von Neumann’s quantum entropy, Shannon’s information theory, Rényi’s generalized entropies, and Kolmogorov–Sinai entropy.

Clausius’s contribution is represented by the integrability relation for reversible heat transfer, $dS=dQ/T$, and by the state-function formulation of classical thermodynamics. The paper emphasizes that Clausius’s global statement that the entropy of the universe “tends towards a maximum” is not directly supported by the precise formalism of equilibrium thermodynamics. This historical qualification is important because the later mathematical treatment does not identify every entropy-like quantity with a universal monotonic thermodynamic state variable.

Boltzmann’s two major contributions are interpreted through empirical measures. In the kinetic theory, a microscopic configuration is mapped to a one-particle distribution; in the combinatorial theory, a configuration in $A^N$ is mapped to its empirical distribution. This common coarse-graining structure connects the 1872 and 1877 approaches and anticipates the modern role of empirical measures in Sanov’s theorem. The type-class cardinality formula

$|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$

yields the asymptotic relation

$\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$

where $S(p)$ is the Boltzmann–Shannon entropy. The implication is that entropy measures the exponential multiplicity of microscopic configurations compatible with a macroscopic empirical distribution.

Gibbs’s formulation is presented through the variational duality between entropy and free energy. For a finite configuration space, the Gibbs distribution uniquely minimizes the free-energy functional, and the resulting identity reproduces the equilibrium relation $F=E-TS$. This establishes convex duality as a common mathematical structure underlying both statistical mechanics and large deviations.

Einstein’s contribution is interpreted as the fluctuation counterpart of Boltzmann’s counting argument. The relation $W=e^{S/k}$ is read as an asymptotic statement about the probability of macroscopic fluctuations. The paper’s later large-deviation framework makes this intuition precise: probabilities of atypical empirical measures decay exponentially, with the relative entropy as the exponent.

The treatment of von Neumann, Shannon, and Rényi entropy culminates in the claim that the central transition in the modern theory is from one-argument entropy to two-argument relative entropy. Shannon’s entropy is characterized axiomatically, but the paper follows Shannon in treating the operational consequences—coding and distinguishability—as more important than the axiomatic characterization itself. Rényi entropy arises by weakening Shannon’s conditional additivity property to additivity under independent products.

Classical entropy and large deviations

The classical theory is developed around the empirical measure

$$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$

For an i.i.d. source with prior $q$, the empirical measure converges almost surely to $q$. Large deviations quantify the exponentially rare events in which $A^N$0 remains away from $A^N$1.

The central rate function is the Kullback–Leibler divergence

$A^N$2

with the convention that it is infinite when $A^N$3 is not absolutely continuous with respect to $A^N$4. Gibbs’s inequality gives non-negativity and identifies $A^N$5 as the unique zero. The paper repeatedly uses this equality condition to connect equilibrium, typicality, and statistical distinguishability.

Sanov’s theorem is the principal result:

$A^N$6

for closed $A^N$7, with the corresponding lower bound for open sets. Thus, the probability of an empirical distribution lying in a set is controlled by the least relative entropy in that set. When the relevant interior and closure infima coincide, the upper and lower bounds yield an exact exponential limit.

Several consequences are established within the same framework.

• Typicality: deviations from $A^N$8 have exponentially small probability. This strengthens the strong law by specifying the exponential rate of convergence.
• Maximum entropy and Gibbs distributions: minimizing relative entropy under an energy constraint produces the Gibbs distribution.
• Fenchel duality: relative entropy and pressure are Legendre–Fenchel duals. The pressure is

$A^N$9

and the relative entropy is recovered as its convex conjugate.

• Statistical mechanics: the Gibbs variational principle and the large-deviation variational principle are manifestations of the same convex duality.

The exposition is strongest when it makes explicit that the entropy functional is not merely a measure of uncertainty. It is the exponential cost of imposing an empirical distribution $|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$0 when the generating distribution is $|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$1.

The asymptotic equipartition property and coding

The asymptotic equipartition property is derived from the weak and strong laws of large numbers. For a typical set $|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$2, the paper establishes

$|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$3

and

$|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$4

Each typical sequence has probability approximately $|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$5, while the typical set carries asymptotically all probability. If $|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$6 is nonuniform, this set is an exponentially small subset of the full sample space $|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$7.

The coding consequences are quantitative. For noiseless binary coding, the optimal asymptotic average length per symbol is $|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$8, while a Shannon prefix code satisfies

$|T_N(p)|=\frac{N!}{\prod_a (Np(a))!}$9

Equality is possible only in the special case that all probabilities have dyadic form. The noisy coding theorem gives the sharp threshold: rates above $\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$0 permit decoding error tending to zero, whereas rates below $\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$1 force the error probability to tend to one. This is a particularly clear example of entropy functioning as an operational threshold rather than merely as a descriptive statistic.

Cramér’s theorem and the general large-deviation framework

Cramér’s theorem is derived as a contraction of Sanov’s theorem. If $\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$2 is an observable, the sample mean

$\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$3

has rate function

$\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$4

The infimum is achieved by a Gibbs distribution tilted by the observable. Equivalently,

$\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$5

where

$\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$6

This result formalizes the maximum-entropy derivation of equilibrium distributions. The rate function is convex, lower semicontinuous, nonnegative, and uniquely minimized at the typical value $\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$7. For a standard Gaussian source, the paper records the explicit rate function $\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$8, giving the asymptotic tail law

$\frac{1}{N}\log |T_N(p)|\longrightarrow S(p),$9

for $S(p)$0. The distinction between $S(p)$1 central-limit fluctuations and $S(p)$2 large deviations is consequently made precise.

The general theory is then organized around the large-deviation principle, consisting of a sequence of probability measures, a lower-semicontinuous rate function, and upper and lower exponential bounds on closed and open sets. The contraction principle explains how rate functions transform under continuous observables. Varadhan’s theorem gives the asymptotics of exponential integrals,

$S(p)$3

and identifies exponential tilting as a transformation of the rate function. Bryc’s theorem and the Gärtner–Ellis theorem provide converse mechanisms: under suitable exponential-moment and differentiability assumptions, pressure determines the rate function.

The assumptions matter. The Gärtner–Ellis presentation requires differentiability of the limiting pressure, and the proof sketch temporarily assumes twice differentiability to obtain a weak law under tilted measures. The text acknowledges that more general versions require convex-analytic arguments not developed in full.

Hypothesis testing and the operational role of relative entropy

The paper uses binary hypothesis testing to give relative entropy its most direct operational meaning. For distributions $S(p)$4 and $S(p)$5, the Neyman–Pearson test compares likelihoods, equivalently comparing the empirical relative entropies

$S(p)$6

Three asymptotic regimes are treated.

Chernoff discrimination

Under symmetric testing, the minimum total error $S(p)$7 decays at the Chernoff rate:

$S(p)$8

where

$S(p)$9

Unlike Kullback–Leibler divergence, the Chernoff entropy is symmetric. It therefore serves as an asymptotically optimal symmetric distinguishability measure.

Stein discrimination

When the type-I error is bounded by a fixed $F=E-TS$0, the optimal type-II error satisfies

$F=E-TS$1

The exponent is independent of $F=E-TS$2. This result explains why relative entropy is more than a generalized distance: it is the optimal exponential decay rate for one-sided statistical discrimination under a nontrivial error constraint.

Hoeffding tradeoff

If the type-I error is required to decay exponentially at rate $F=E-TS$3, then the optimal type-II exponent is governed by a Hoeffding divergence. The paper identifies a threshold at

$F=E-TS$4

Below this threshold, the type-II error still decays exponentially; at and above it, the type-II error converges to one. Thus, over-constraining one error probability destroys the performance of the other. This result makes explicit the asymmetric tradeoff that is hidden by the symmetric Chernoff formulation.

Quantum entropy and hypothesis testing

The quantum part begins in finite-dimensional Hilbert spaces, where density operators provide a direct analogue of classical probability distributions. Umegaki relative entropy is

$F=E-TS$5

with value $F=E-TS$6 when the support condition fails.

Quantum Stein’s lemma gives the exact analogue of the classical result:

$F=E-TS$7

The operational interpretation is retained despite noncommutativity: the quantum relative entropy is the optimal asymptotic exponent for discriminating $F=E-TS$8 from $F=E-TS$9 under a fixed type-I constraint.

The discussion of entanglement illustrates the difference between classical and quantum entropy. A pure bipartite state has zero global von Neumann entropy, but its reduced state can have positive entropy. For Bell states, the reduced density matrix is maximally mixed and the entanglement entropy is $W=e^{S/k}$0. This is a sharp structural contrast with classical probability, in which a pure point measure always has zero entropy under marginalization.

The paper also discusses composite hypothesis testing against the convex set of separable states. For an entangled state $W=e^{S/k}$1, the asymptotic exponent is governed by the minimum quantum relative entropy from $W=e^{S/k}$2 to the separable set. The result has the formal appearance of a large-deviation principle, but the notes correctly emphasize that it is a statement about optimal quantum measurements and hypothesis-testing errors, not a direct quantum analogue of Sanov’s theorem. The paper makes the deliberately strong claim that many results called “quantum Sanov theorems” are actually quantum versions of Stein’s lemma and should not be identified with a full noncommutative empirical-measure large-deviation theorem.

Von Neumann algebras and modular theory

The central technical problem in the infinite-system setting is that a state on a von Neumann subalgebra $W=e^{S/k}$3 can have many density operators on the ambient Hilbert space. These representatives need not have the same entropy. Consequently, the ordinary trace formula for von Neumann or Umegaki entropy is not intrinsic to the subsystem.

This is not a minor technicality. The paper gives an entangled-state example in which one ambient density operator representing the same restricted state has zero entropy, while another representative yields the physically correct reduced-state entropy. Therefore, density matrices in the ambient algebra cannot be used naively to define subsystem entropy.

Tomita–Takesaki theory resolves the structural problem. Given a cyclic and separating vector $W=e^{S/k}$4 for a von Neumann algebra $W=e^{S/k}$5, the antilinear operator

$W=e^{S/k}$6

has polar decomposition

$W=e^{S/k}$7

The modular automorphism group

$W=e^{S/k}$8

defines a canonical dynamics of $W=e^{S/k}$9, and the modular conjugation implements the relation

$$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$0

The physical and mathematical perspectives are shown to be equivalent. In finite systems, a thermal state determines the dynamics through the Hamiltonian and satisfies the KMS condition. In the operator-algebraic formulation, a faithful state determines the modular dynamics, and the resulting state-dynamics pair satisfies the KMS condition. This equivalence is the conceptual bridge between equilibrium statistical mechanics and abstract operator algebras.

For two faithful states, relative modular operators yield Araki relative entropy:

$$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$1

In type I algebras this reduces to Umegaki relative entropy; in commutative algebras it reduces to Kullback–Leibler divergence. Thus Araki’s construction simultaneously extends both classical and finite-dimensional quantum relative entropy.

Haagerup $$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$2 spaces and intrinsic density operators

Modular theory defines the appropriate relative modular objects, but the paper argues that computation requires Haagerup’s noncommutative $$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$3 spaces. These spaces generalize the usual $$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$4 spaces and the Schatten ideals.

The relevant structural correspondences are:

• $$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$5 is identified with $$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$6;
• $$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$7 is identified with the predual $$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$8;
• $$L_N(\sigma)=\frac{1}{N}\sum_{n=0}^{N-1}\delta_{\sigma_n}.$$9 is a Hilbert space in standard form;
• normal states correspond canonically to positive elements of $q$0.

In this framework, each normal state has an intrinsic density operator in $q$1, even when the algebra is not type I and has no ordinary trace. Relative entropy can then be written formally as

$q$2

where $q$3 are the canonical Haagerup densities. The trace is not the ordinary Hilbert-space trace; it is constructed through the crossed product of $q$4 by its modular action.

This is the paper’s principal mathematical synthesis: classical densities, finite-dimensional density matrices, and densities of states on arbitrary von Neumann algebras become instances of a single noncommutative $q$5 formalism. The construction is technically demanding and involves unbounded operators, but it removes the nonuniqueness of ambient density representatives that otherwise obstructs intrinsic entropy.

Dynamical entropy

The notes also cover entropy in classical and quantum dynamical systems. Kolmogorov–Sinai entropy is defined from the growth rate of the Shannon entropy of increasingly refined trajectory partitions:

$q$6

followed by a supremum over finite measurable partitions. This entropy measures the asymptotic information production of a measure-preserving transformation and connects to dynamical invariants such as positive Lyapunov exponents through Pesin’s formula.

Its quantum counterpart is the Connes–Størmer–Narnhofer–Thirring entropy for automorphism systems of von Neumann algebras. The paper does not develop this theory to the same depth as large deviations or modular entropy, but includes it to show that entropy also quantifies dynamical complexity, not only equilibrium fluctuation or state distinguishability.

Limitations and open questions

The paper is intentionally a course rather than a research monograph. Many results are proved only in finite-state settings and then stated for Polish spaces or general von Neumann algebras. The proofs of several technically central facts—including the full Polish-space versions of Sanov’s theorem, the most general Gärtner–Ellis theorem, and parts of Hoeffding theory—are omitted or only sketched.

The quantum-classical analogy is also carefully limited. The notes provide strong quantum analogues of hypothesis-testing results, but they do not produce a general quantum empirical-measure large-deviation theory that would parallel Sanov’s theorem. The paper explicitly leaves open the question of what, in full noncommutative generality, should replace the unifying role played by classical large deviations.

Other stated omissions are the second law of thermodynamics, entropy methods for PDEs, and black-hole thermodynamics. These are not treated as consequences of the formalism developed in the lectures. In particular, the text does not claim that relative entropy alone resolves the conceptual difficulties surrounding thermodynamic entropy or black-hole entropy.

Conclusion

The paper’s principal achievement is expository and structural. It places Boltzmann, Shannon, Gibbs, Rényi, von Neumann, Umegaki, Araki, and Kolmogorov–Sinai entropies within a common framework centered on relative entropy, convex duality, asymptotic probability, and operator algebras. On the classical side, Sanov’s and Cramér’s theorems explain entropy as an exponential fluctuation cost. In hypothesis testing, the same relative entropies become optimal error exponents. On the quantum side, modular theory and Haagerup $q$7 spaces provide the intrinsic constructions required for infinite systems and non-type-I subsystems. The resulting account is technically broad, explicit about its assumptions, and organized around the claim that entropy is best understood through the operational and asymptotic structures it governs.