CNEMF-MDPs: Conditional Non-Exchangeable MDPs

• The framework defines CNEMF-MDPs as infinite-horizon MDPs that lift agent-level dynamics to a measure-valued state space, uniquely characterizing the value function through a fixed-point Bellman operator.
• It establishes both strong and weak (label-state) formulations with randomized open-loop controls to capture agent heterogeneity and conditional dynamics without assuming exchangeability.
• The approach integrates neural operator learning via DeepONet-type architectures to approximate infinite-dimensional Bellman operators, offering concrete computational methods for high-dimensional control problems.

Conditional Non-Exchangeable Mean Field MDPs (CNEMF-MDPs) are infinite-horizon Markov Decision Processes with a continuum of heterogeneous agents interacting through a common noise, without assuming exchangeability. The framework is introduced in both a strong formulation and a label-state formulation, and the control problem is lifted to a standard MDP defined on the Wasserstein space of probability measures over the product of the label and state spaces. In this formulation, the label space represents agent heterogeneity, the state space is the individual state space, and a fixed distribution specifies the population of agent labels. Within this lifted state space, the value function is characterized as the unique fixed point of an appropriate Bellman operator. Subsequent work studies operators on constrained measure spaces of labeled conditional distributions and uses DeepONet-type branch-trunk architectures to approximate the resulting infinite-dimensional Bellman operators, thereby connecting CNEMF-MDPs to neural operator learning on $\mathcal M_\lambda$ (Mekkaoui et al., 26 Jan 2026, Mekkaoui et al., 23 Mar 2026).

1. Strong and weak formulations

The canonical CNEMF-MDP setup begins with a label space $I=[0,1]$ endowed with a fixed (Lebesgue) law $\lambda$, an individual state space $X$ that is a compact Polish space with metric $d$, and an action space $A$ that is compact Polish with metric $d_A$. Heterogeneity is encoded by the label $u\in I$, and the population is not assumed to be exchangeable because labels enter the dynamics and reward explicitly. The stochastic environment contains idiosyncratic noises $\varepsilon_t^u$, $u\in I$, i.i.d. with law $I=[0,1]$0, and common noise $I=[0,1]$1, i.i.d. with law $I=[0,1]$2, independent of the idiosyncratic noises. Initial information is represented by $I=[0,1]$3, independent across $I=[0,1]$4 and from the noises, with $I=[0,1]$5 (Mekkaoui et al., 26 Jan 2026).

In the strong formulation, admissible controls are randomized open-loop families $I=[0,1]$6 indexed by $I=[0,1]$7 and $I=[0,1]$8,

$I=[0,1]$9

where $\lambda$0 are independent randomizers. The state dynamics are

$\lambda$1

with conditional joint law

$\lambda$2

The corresponding discounted reward is

$\lambda$3

The weak, or label-state, formulation introduces a representative label $\lambda$4 and a single process $\lambda$5. One writes

$\lambda$6

with admissible controls

$\lambda$7

and payoff

$\lambda$8

A key theorem shows that $\lambda$9 whenever the laws of $X$0 and $X$1 coincide. This establishes equivalence between the strong and weak formulations and identifies the control problem with a law-invariant object indexed by the initial label-state distribution (Mekkaoui et al., 26 Jan 2026).

2. Lifted Wasserstein state space and Bellman equation

The defining structural move in CNEMF-MDPs is the lift from agent-level dynamics to a measure-valued MDP. In the general formulation, the lifted state is

$X$2

and the lifted control is

$X$3

The transition kernel is

$X$4

so that $X$5. The stage cost is

$X$6

and the infinite-horizon value satisfies

$X$7

The Bellman operator on bounded measurable $X$8 is

$X$9

Under Lipschitz-type assumptions on $d$0 and $d$1 in $d$2, $d$3 is a $d$4-contraction in sup-norm, so the value function is the unique fixed point $d$5 (Mekkaoui et al., 26 Jan 2026).

A more specialized representation appears when the individual state space is $d$6 and the first marginal is prescribed. Let $d$7 be a label space endowed with a reference measure $d$8, and define

$d$9

By disintegration,

$A$0

so $A$1 is the conditional law of the $A$2-agent. In this form, a stationary Markov feedback is a measurable map $A$3, and the transition operator $A$4 is given by

$A$5

Equivalently,

$A$6

The reward functional is

$A$7

and the Bellman equation takes the form

$A$8

or, in finite horizon,

$A$9

This measure-valued Bellman equation makes explicit that the state variable is the full labeled conditional distribution, not a symmetric empirical law (Mekkaoui et al., 23 Mar 2026).

3. Disintegration, randomization, and measurable selection

The non-exchangeable character of CNEMF-MDPs is encoded by the label marginal constraint and by disintegration. Elements of $d_A$0 correspond to families of labeled conditional distributions $d_A$1 through

$d_A$2

This is more restrictive than an arbitrary probability measure on $d_A$3 and more informative than an exchangeable law on states alone, because the prescribed first marginal keeps the population of labels fixed while allowing label-dependent state laws and controls. A plausible implication is that the mean field in a CNEMF-MDP is best viewed as a conditional family indexed by labels rather than as a single unlabeled population distribution (Mekkaoui et al., 23 Mar 2026).

Randomization is structurally important. In the precursor CMKV-MDP analysis with common noise and open-loop controls, the lifted MDP on the space of probability measures requires relaxed controls of the form

$d_A$4

or, equivalently, actions in $d_A$5. Because the planner must “randomize” over the population to match an arbitrary $d_A$6 when the $d_A$7-field on the population is not already rich, one must allow randomized feedback policies

$d_A$8

That analysis also proves a measurable optimal coupling lemma: for compact Polish $d_A$9 there exists

$u\in I$0

measurable such that if $u\in I$1 and $u\in I$2 independent of $u\in I$3, then $u\in I$4 satisfies $u\in I$5 and

$u\in I$6

This measurable-coupling construction is the technical device used to obtain jointly measurable selections and $u\in I$7-optimal randomized feedback controls (Motte et al., 2019).

The CNEMF-MDP framework inherits this emphasis on randomization. In the 2026 formulation, admissible controls are randomized open-loop, and from an $u\in I$8-optimal lifted randomized feedback $u\in I$9 one constructs strong and weak randomized feedback controls achieving the same performance. In the related non-exchangeable mean-field control problem with controlled interactions, relaxed admissible laws are the closure, in the Wasserstein topology, of strong feedback-law-valued pairs, and one has

$\varepsilon_t^u$0

together with existence of an optimal relaxed control under mild continuity and Lipschitz assumptions on $\varepsilon_t^u$1 (Mekkaoui et al., 26 Jan 2026, Djete, 31 Oct 2025).

4. Quantitative propagation of chaos and finite-population control

A major contribution of the CNEMF-MDP theory is a quantitative comparison between the lifted infinite-agent problem and a finite $\varepsilon_t^u$2-agent MDP with common noise. In the finite system, the state is $\varepsilon_t^u$3, the control is $\varepsilon_t^u$4, and transitions are

$\varepsilon_t^u$5

with empirical law

$\varepsilon_t^u$6

The reward is

$\varepsilon_t^u$7

and the Bellman operator on $\varepsilon_t^u$8 is

$\varepsilon_t^u$9

Under parallel Lipschitz assumptions, $u\in I$0 is a $u\in I$1-contraction with fixed point $u\in I$2 (Mekkaoui et al., 26 Jan 2026).

To compare finite and infinite systems, for any lifting $u\in I$3 one defines

$u\in I$4

where $u\in I$5 is the labeled empirical measure sending $u\in I$6 to $u\in I$7. The finite-infinite comparison yields, for matching feedbacks,

$u\in I$8

where

$u\in I$9

By choosing a minimizing coupling, one deduces the explicit quantitative bound

$I=[0,1]$00

The same analysis yields a near-optimal policy construction: from an $I=[0,1]$01-optimal lifted feedback $I=[0,1]$02 one builds an $I=[0,1]$03-agent feedback

$I=[0,1]$04

producing an

$I=[0,1]$05

-optimal $I=[0,1]$06-agent policy (Mekkaoui et al., 26 Jan 2026).

These estimates formalize the propagation of chaos in a non-exchangeable setting with common noise. They also justify the mean-field approximation operationally: $I=[0,1]$07-optimal policies for the limit CNEMF-MDP can be converted into near-optimal policies for finite heterogeneous systems. The paper identifies large-scale networked systems with heterogeneity, including power grids, financial networks, and graphon games, as application domains for this finite-to-infinite control transfer (Mekkaoui et al., 26 Jan 2026).

5. Neural operator approximation on $I=[0,1]$08

The operator-learning perspective treats the Bellman map and related value operators as functions on the constrained measure space $I=[0,1]$09. The central approximation theorem states, in paraphrased form, that if

$I=[0,1]$10

is continuous with mild growth and $I=[0,1]$11 is any probability law over $I=[0,1]$12, then for every $I=[0,1]$13 there exist $I=[0,1]$14, cylindrical features $I=[0,1]$15, a branch network $I=[0,1]$16, and trunk networks $I=[0,1]$17 such that, with

$I=[0,1]$18

one has

$I=[0,1]$19

The proof combines cylindrical approximations of probability measures with a DeepONet-type branch-trunk neural architecture. Its two stated ingredients are: first, cylindrical approximation on compact subsets of $I=[0,1]$20 by functions of the form

$I=[0,1]$21

and second, finite-dimensional universal approximation of each $I=[0,1]$22 and $I=[0,1]$23 by small feed-forward nets (Mekkaoui et al., 23 Mar 2026).

In practical form, the learned operator is written

$I=[0,1]$24

or equivalently

$I=[0,1]$25

with $I=[0,1]$26 and $I=[0,1]$27. The trunk net takes $I=[0,1]$28 as input and outputs a scalar basis function, while the branch net takes the $I=[0,1]$29-vector of moments $I=[0,1]$30 and outputs the coefficients. This is the specific DeepONet branch-trunk architecture proposed for conditional mean-field operators on labeled distributions (Mekkaoui et al., 23 Mar 2026).

The sampling strategy for training is explicit. One fixes a non-atomic reference $I=[0,1]$31, samples a random transport map $I=[0,1]$32 in some parametrized family so that for each $I=[0,1]$33, $I=[0,1]$34 is the desired $I=[0,1]$35, draws i.i.d. samples $I=[0,1]$36, and sets $I=[0,1]$37. The empirical law of $I=[0,1]$38 then lies in $I=[0,1]$39. Repeating this procedure yields training measures $I=[0,1]$40, and targets can be computed either by solving a reference discretization of the Bellman equation, or by Monte-Carlo simulation followed by regression. As a proof of concept, the paper applies this machinery to a CNEMF-MDP with controlled SDE / Euler-scheme dynamics and approximates the Bellman operator

$I=[0,1]$41

Training uses $I=[0,1]$42 sampled laws $I=[0,1]$43, and for each $I=[0,1]$44, $I=[0,1]$45 particles to estimate temporal or dynamic-programming targets. The reported diagnostics are mean-squared rollout error versus the number $I=[0,1]$46 of trunk-branch sensors, error as a function of the number $I=[0,1]$47 of moments, and computational times on a single GPU. In all cases the experiments observe rapid decay of approximation error as $I=[0,1]$48 increase, confirming the universal-approximation theory in practice (Mekkaoui et al., 23 Mar 2026).

6. Controlled interactions, related models, and interpretive issues

A related non-exchangeable mean-field control problem extends the CNEMF viewpoint by making the interaction structure itself controllable. In that formulation, each agent carries a private state $I=[0,1]$49 and a label $I=[0,1]$50, and there are two kinds of controls: a local action $I=[0,1]$51 and a pairwise interaction action $I=[0,1]$52. Heterogeneity is encoded by a structural kernel

$I=[0,1]$53

typically of graphon type. In the finite-$I=[0,1]$54 system, this kernel is approximated by a step-kernel

$I=[0,1]$55

which converges in cut-norm to $I=[0,1]$56. The finite-agent dynamics are McKean-Vlasov-type SDEs involving empirical measures $I=[0,1]$57 and $I=[0,1]$58 that record outgoing and incoming controlled interactions, while the mean-field limit replaces sums by integrals and empirical laws by a flow $I=[0,1]$59 (Djete, 31 Oct 2025).

This controlled-interaction model proves several properties that align with the broader CNEMF program. Under mild continuity and Lipschitz assumptions on $I=[0,1]$60, the set of relaxed admissible laws is compact and convex in law, relaxed and strong formulations coincide in value, and an optimal relaxed control exists. If $I=[0,1]$61 for all bounded Lipschitz $I=[0,1]$62, then asymptotically optimal finite-$I=[0,1]$63 controls admit subsequences whose empirical laws converge weakly to relaxed mean-field controls, and conversely any Lipschitz mean-field control can be lifted to finite systems with convergent empirical laws and payoffs. Together these imply

$I=[0,1]$64

as well as convergence of $I=[0,1]$65-optimal $I=[0,1]$66-player controls to mean-field optimal controls. The corresponding Bellman-type equation is written on the space of measures and kernels:

$I=[0,1]$67

Under standard Lipschitz and boundedness assumptions on $I=[0,1]$68 and compactness of $I=[0,1]$69, $I=[0,1]$70 is continuous in $I=[0,1]$71 in Wasserstein distance and in $I=[0,1]$72 in cut-norm (Djete, 31 Oct 2025).

Several interpretive points follow directly from these results. A common misconception is that heterogeneity can be absorbed into the state while preserving the standard exchangeable mean-field structure. The CNEMF-MDP framework is introduced precisely for systems “without assuming exchangeability,” and the fixed label distribution is part of the state constraint in $I=[0,1]$73 or $I=[0,1]$74 (Mekkaoui et al., 26 Jan 2026). Another misconception is that pure deterministic feedback always suffices. The common-noise mean-field MDP literature emphasizes the crucial role of relaxed controls and randomization hypotheses, and the non-exchangeable extensions retain that feature through lifted randomized feedbacks and relaxed law-valued controls (Motte et al., 2019). Finally, the cited work identifies continuous-time extensions, learning in CNEMF-MDPs, and relaxed controls under weaker regularity as future directions, while the operator-learning results suggest a concrete computational route for high-dimensional Bellman problems on constrained Wasserstein spaces (Mekkaoui et al., 26 Jan 2026, Mekkaoui et al., 23 Mar 2026).