SoftFair: Soft Fairness via Softmax VI

• The paper introduces a model-based resource allocation algorithm that enforces soft fairness in finite-horizon RMAB problems while nearly achieving optimal rewards.
• It utilizes a softmax value-iteration process with per-arm advantage computations to ensure that arms with higher long-term value are favored under resource constraints.
• Empirical evaluations on both real and synthetic data demonstrate that SoftFair effectively balances intervention benefit and fairness with strong theoretical guarantees and computational efficiency.

Soft Fairness via Softmax Value-Iteration (SoftFair) is a model-based resource allocation algorithm tailored for finite-horizon restless multi-armed bandit (RMAB) problems. It encodes a soft notion of fairness—ensuring no arm with lower long-term value is probabilistically favored over a better one—while maintaining high overall reward. SoftFair utilizes a softmax value-iteration process to select multi-armed intervention policies under stringent resource constraints, with strong theoretical guarantees regarding both fairness and near-optimality. The design is motivated by real-world settings demanding balanced allocation, such as public health interventions, where repeated deprivation of certain communities can result in inequity and suboptimal outcomes (Li et al., 2022).

1. Restless Multi-Armed Bandits: Framework and Notation

The RMAB framework considers $N$ independent two-state arms (indexed by $i = 1, \dots, N$), each described as a Markov Decision Process (MDP):

• State space: $S_i = \{0, 1\}$, with $1$ the "good" state and $0$ the "bad" state.
• Action space: $A_i = \{0, 1\}$; $a_i = 1$ denotes intervention ("active"), $a_i = 0$ means no intervention ("passive").
• Transition probabilities: $$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$.
• Immediate reward: $R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$ (probability of transitioning to the good state).

At each time step $i = 1, \dots, N$0, with finite $i = 1, \dots, N$1, the joint state $i = 1, \dots, N$2 is observed, and an action vector $i = 1, \dots, N$3 is chosen, subject to the resource constraint

$i = 1, \dots, N$4

The immediate reward obtained is $i = 1, \dots, N$5.

A stochastic policy $i = 1, \dots, N$6 is a collection of distributions $i = 1, \dots, N$7 for $i = 1, \dots, N$8, with $i = 1, \dots, N$9. The expected discounted cumulative reward from $S_i = \{0, 1\}$0 at time $S_i = \{0, 1\}$1 under $S_i = \{0, 1\}$2 is $S_i = \{0, 1\}$3, and the value function is $S_i = \{0, 1\}$4. Optimal quantities are $S_i = \{0, 1\}$5.

2. Soft Fairness: Constraint Definition

Soft fairness is formalized by insisting that a stochastic policy $S_i = \{0, 1\}$6 never assigns higher probability to a suboptimal action:

Soft Fairness Definition: A policy $S_i = \{0, 1\}$7 is soft-fair if, for all $S_i = \{0, 1\}$8, for any two joint actions $S_i = \{0, 1\}$9 and observed state $1$0,

$1$1

Thus, no action is more probable unless its optimal long-term cumulative value is at least as high. This soft form avoids starvation and supports fairness across all arms, communities, or regions when allocating limited interventions (Li et al., 2022).

3. SoftFair Algorithm: Principles, Computation, and Procedure

SoftFair operationalizes soft fairness by leveraging per-arm advantage computation, softmax action selection, and value-iteration-based updates.

3.1 Per-Arm Advantage and Logits

For each arm at episode $1$2 and time $1$3, the algorithm computes:

$1$4

The "advantage-like" quantity is

$1$5

with arm logit

$1$6

3.2 Softmax Selection for Resource Constraints

For each arm in the observed multimodal state $1$7, compute the logits $1$8. Select $1$9 arms by drawing without replacement according to softmax-normalized logits:

$0$0

where $0$1 is the inverse temperature coefficient, mediating the trade-off between exploration (fairness) and exploitation (reward).

3.3 Bellman-Style Value Iteration

After selecting the arms and observing transitions:

$0$2

Only visited states are updated, and $0$3 can be updated similarly.

3.4 Summary Pseudocode

The procedure iterates for $0$4 episodes, $0$5 time steps, and $0$6 arms per step, with softmax-based selection and Bellman updates at each round. The final estimate $0$7 is used to extract a stationary policy.

4. Theoretical Guarantees

4.1 Asymptotic Optimality

As $0$8, SoftFair reduces to greedy selection of the $0$9 arms with maximal single-step advantage $A_i = \{0, 1\}$0. This recovers the deterministic optimal $A_i = \{0, 1\}$1-arm policy: maximizing

$A_i = \{0, 1\}$2

thus maximizing the joint action-value $A_i = \{0, 1\}$3 among all feasible $A_i = \{0, 1\}$4 (Li et al., 2022).

4.2 Fairness Enforcement

For any finite $A_i = \{0, 1\}$5, SoftFair’s action selection probability $A_i = \{0, 1\}$6 is monotonic in the joint $A_i = \{0, 1\}$7 of the selected arms’ logits. Therefore, $A_i = \{0, 1\}$8 only if $A_i = \{0, 1\}$9. Smaller $a_i = 1$0 induces higher entropy and more balanced (fair) outcomes, while larger $a_i = 1$1 approaches greedy selection.

4.3 Performance Bounds

Compared to the optimal Bellman operator $a_i = 1$2, the “softness gap” $a_i = 1$3 is bounded:

$a_i = 1$4

This gap decays with increasing $a_i = 1$5. Long-horizon value function convergence is bounded by

$a_i = 1$6

analogously for $a_i = 1$7.

5. Computational Complexity and Implementation Characteristics

For every episode and time step:

• Computing two $a_i = 1$8-values per arm and their corresponding logits: $a_i = 1$9.
• Sampling $a_i = 0$0 arms using softmax without replacement (sequentially): $a_i = 0$1 (naïve method).
• Value-function updates: $a_i = 0$2 per time step.
• Overall computational complexity: $a_i = 0$3 for $a_i = 0$4 episodes, $a_i = 0$5 time steps, and $a_i = 0$6 arms.
• Memory usage: $a_i = 0$7 to store value functions.

SoftFair places no indexability constraints, unlike Whittle-index methods, and is naturally applicable to finite-horizon settings.

6. Experimental Evaluation

SoftFair was evaluated on two settings: a real continuous positive airway pressure (CPAP) adherence dataset ($a_i = 0$8) and a synthetic two-state model with structured transitions. The main baselines for comparison include:

• Random selection,
• Myopic (pick $a_i = 0$9 arms with highest single-step gain $$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$0),
• FairMyopic (softmax on $$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$1),
• Oracle (finite-horizon Whittle-approximation).

Metrics reported included Intervention Benefit $$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$2 and Action Entropy $$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$3, where $$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$4 is the normalized frequency arm $$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$5 is chosen.

Policy	Benefit (%)	Action Entropy
Random	$$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$6	$$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$7
Myopic	$$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$8	$$P^a_{s, s'} = \Pr(s_{i, t+1} = s' \mid s_{i, t} = s, a_{i, t} = a)$$9
FairMyopic	$R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$0	$R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$1
SoftFair ($R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$2)	$R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$3	$R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$4

Compared to the myopic policy (which yields the highest reward but lowest fairness), SoftFair retains most of the intervention benefit while markedly improving action entropy (fairness). On synthetic data, SoftFair consistently outperformed Random and FairMyopic and closely tracked Oracle performance as $R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$5 and $R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$6 increased. Varying the softmax parameter $R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$7 enables controlled interpolation between fairness and optimality; larger $R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$8 approaches the greedy regime at the cost of fairness (lower entropy), smaller $R_i(s, a) = \Pr(s' = 1 \mid s, a) \in [0, 1]$9 gives more uniform (fairer) distributions.

7. Conclusion and Significance

SoftFair delivers a principled, computationally tractable approach to imposing soft fairness in RMABs via softmax value iteration, with explicit mechanisms to mediate the reward–fairness trade-off through a single parameter $i = 1, \dots, N$00. It applies to non-indexable, finite-horizon RMABs and admits rigorous theoretical guarantees on both fairness and performance, as well as practical, empirically validated efficacy in both real and simulated domains (Li et al., 2022).