MARL-Optimized Reviewer Assignment

• The paper introduces a Constrained Multi-Agent Reinforcement Learning framework that optimizes reviewer assignments by balancing timeliness, review quality, and group fairness.
• MARL-Optimized Reviewer Assignment models the process as a stochastic multi-agent game, incorporating dynamic states like reviewer load, historical lateness, decline rates, and topic distances.
• The approach combines rigorous reward design, trust-region policy updates, and an ILP-based runtime mechanism to ensure efficient, fair, and computationally tractable assignments.

MARL-Optimized Reviewer Assignment is a computational framework for peer review assignment based on Constrained Multi-Agent Reinforcement Learning (MARL), designed to address problems of reviewer fatigue, fairness, and review quality in the peer review ecosystem. It models the assignment process as a stochastic multi-agent game, where agents (reviewers) interact with a dynamically evolving system representing real-world constraints and objectives. This method seeks not only to optimize the matching of papers to reviewers, but also to enforce explicit constraints on timeliness, load-balance, and group equity, while adapting to historical reviewer behaviors and institutional demands (Farooq et al., 27 Jan 2026).

1. Formal Multi-Agent Model for Reviewer Assignment

In this formulation, the reviewer assignment subproblem is codified as a constrained stochastic game $$G = \langle \mathcal{A}, \mathcal{S}, \mathcal{A}_c, \mathcal{P}, \mathcal{R}, \mathcal{C} \rangle$$:

• Agents $\mathcal{A}$: Each reviewer is an agent $i \in \{1,\ldots,N\}$.
• State Space $\mathcal{S}$: At each assignment epoch $t$, the state $s_t$ aggregates reviewer-level features:
  • $ℓ_i^t$ (current uncompleted load for reviewer $i$)
  • $τ_i^t$ (historical mean lateness of reviewer $i$)
  • $\mathcal{A}$0 (decline rate of reviewer $\mathcal{A}$1)
  • $\mathcal{A}$2 (topic distance between reviewer $\mathcal{A}$3 and paper $\mathcal{A}$4)
  • $\mathcal{A}$5 with $\mathcal{A}$6 for $\mathcal{A}$7 papers.
• Joint Action $\mathcal{A}$8: Assignment matrix $\mathcal{A}$9 with $i \in \{1,\ldots,N\}$0 iff reviewer $i \in \{1,\ldots,N\}$1 is assigned paper $i \in \{1,\ldots,N\}$2 at epoch $i \in \{1,\ldots,N\}$3.
• Constraints:
  • Exactly $i \in \{1,\ldots,N\}$4 reviewers per paper ($i \in \{1,\ldots,N\}$5)
  • Load-bound per reviewer ($i \in \{1,\ldots,N\}$6)
  • Conflict-of-interest exclusion ($i \in \{1,\ldots,N\}$7 if COI)
• Transition Kernel $i \in \{1,\ldots,N\}$8: $i \in \{1,\ldots,N\}$9 is updated by tracking assignment completion, lateness, and decline.
• Reward $\mathcal{S}$0: Vector per epoch incorporates timeliness, review quality, and fairness penalties. The global instantaneous reward is:

$\mathcal{S}$1

where: - $\mathcal{S}$2: Timeliness indicator for reviewer $\mathcal{S}$3 - $\mathcal{S}$4: Review specificity/quality score - $\mathcal{S}$5: Group fairness penalty

• Cost Functions $\mathcal{S}$6: Quantities (e.g., load-balance, group-imbalance) that serve as optimization constraints.

2. Reward Design and Constraint Specification

The multi-objective reward is constructed to align incentives and enforce operational constraints:

• Timeliness: $\mathcal{S}$7 if reviewer $\mathcal{S}$8 completes on time, else $\mathcal{S}$9.
• Review Quality: $t$0, typically normalized specificity or meta-review consistency.
• Fairness Penalty: For demographic groups set $t$1, loading per group $t$2 is $t$3; the group fairness penalty is:

$t$4

with $t$5.

The constrained RL objective is:

$t$6

where $t$7 and $t$8 similarly aggregates costs for constraint $t$9. Hard thresholds $s_t$0 are selected (e.g., coefficient of variation of load $s_t$1, group-imbalance $s_t$2).

3. MARL Policy Learning and Offline (Counterfactual) Evaluation

The training approach leverages Constrained Multi-Agent Policy Optimization (CMAPO), extending single-agent constrained policy optimization to multi-agent systems:

• Actor-Critic Architecture: Individual actor networks $s_t$3 (per reviewer); centralized critic $s_t$4 for rewards and costs.
• Trust-Region Policy Update: At each iteration, gradients $s_t$5, $s_t$6 are computed for each agent; a quadratic program solves for joint trust-region updates under all cost constraints:
  • Quadratic program:

  $s_t$7

  where $s_t$8 is the trust-region radius.

• Offline RL with Doubly Robust Estimation: Policy improvement utilizes logged historical data and Doubly Robust off-policy estimators as in Thomas & Brunskill (2016):

$s_t$9

with importance weights $ℓ_i^t$0.

• Hyperparameters:
  • Discount factor: $ℓ_i^t$1
  • Trust-region radius: $ℓ_i^t$2
  • Cost limits: Set to match target CV(load) or group-imbalance limits
  • Batch size: $ℓ_i^t$3 assignment epochs per iteration

Convergence is determined by stabilization of the objective $ℓ_i^t$4 and satisfaction of all cost constraints within specified slack.

4. Runtime Assignment Mechanism and Integer Programming

At test/deployment, the learned MARL policy $ℓ_i^t$5 produces either a probability mask $ℓ_i^t$6, with $ℓ_i^t$7, or direct deterministic assignments.

Given the combinatorial constraints, final reviewer assignment is produced by solving a small-scale Integer Linear Program (ILP):

• Objective: maximize $ℓ_i^t$8 over $ℓ_i^t$9
• Subject to:
  • $i$0 for each paper $i$1
  • $i$2 for each reviewer $i$3
  • $i$4 if conflict-of-interest

Typically, $i$5 or $i$6. This ILP is computationally efficient, solvable in milliseconds for $i$7, $i$8, using solvers such as Gurobi or COIN-OR.

5. Empirical Evaluation, Metrics, and Baselines

Performance assessment is scheduled in "shadow mode" using an Agent-Based Model (ABM) environment instantiated with historical OpenReview data distributions.

• Baselines:
  • ILP topic-matching/min-cost-flow (Charlin & Zemel 2013)
  • Greedy smallest-load assignment
  • Random matching under constraints
• Metrics:
  • Timeliness: Percentage of reviews completed by deadline
  • Load-balance: Coefficient of Variation $i$9
  • Group Fairness: Jain's index: $τ_i^t$0
  • Quality Uplift: $τ_i^t$1

Example metric comparison:

Method	Timeliness	CV(load)	Jain-Fairness	ΔQ
ILP baseline	75%	0.28	0.82	0.00
MARL assign	87%	0.15	0.93	+0.07

Significance is tested via paired $τ_i^t$2-test ($τ_i^t$3).

6. Equity, Threat Models, and Constraint Enforcement

The architecture incorporates explicit fairness and robustness strategies:

• Bias amplification is addressed through fairness constraints in both the reward function ($τ_i^t$4 penalty) and cost terms $τ_i^t$5. Offline simulations enable bias testing.
• Strategic declines by reviewers are mitigated by feeding decline patterns $τ_i^t$6 into the state representation, thereby dynamically down-weighting over-decliners’ assignment likelihood.
• PAC-style risk bounds can be derived for group imbalance: with high probability ($τ_i^t$7), the learned policy ensures group-imbalance $τ_i^t$8 if cost constraints $τ_i^t$9 and estimation errors are sufficiently bounded.

7. Pilot Implementation and Practical Deployment

• Shadow Mode Pilot (2027–2028): Planned trial at ICSE workshop scale, utilizing historical data for simulation and policy rollouts. Implementation in Python + PyTorch, with assignment-phase inference latency of $i$00.5 s for $i$1, $i$2. ILP postprocessing via Gurobi or COIN-OR.
• Compute Requirements: Training requires approximately two weeks on an 8-GPU server for 2000 ABM episodes.
• Operational Targets:
  • Timeliness uplift $i$310%
  • $i$4
  • Jain-fairness $i$5
• Live Mode (2028–2029): Expansion to opt-in live assignment, Program-Chair oversight, manual audit of 10% assignments, and continuous monitoring of group-Gini coefficients.

These advances collectively realize a MARL-driven reviewer matching system that (1) adapts to reviewer fatigue and historical participation patterns, (2) enforces equity and operational constraints by design, (3) learns efficiently from historical data via counterfactual evaluation, and (4) remains computationally tractable for deployment in real-world conference settings (Farooq et al., 27 Jan 2026).