Programmatic Iterated Best Response (PIBR)

• PIBR is an iterative framework that computes equilibrium strategies in multi-agent systems by optimizing each agent's policy against fixed opponent strategies.
• It employs modular subroutines like LLM-based synthesis, linear minimization oracles, and convex optimization to derive interpretable policy representations.
• PIBR offers improved convergence, transparency, and theoretical guarantees over black-box models, facilitating robust strategic decision making across diverse domains.

Programmatic Iterated Best Response (PIBR) is a family of algorithmic frameworks for equilibrium computation and policy synthesis in multi-agent settings, where each agent’s strategy is iteratively optimized in response to the (fixed) strategies of its opponents. The “programmatic” aspect refers either to the explicit representation of policies in a human-interpretable, executable form (e.g., source code), or—more broadly—to architectures where policies, strategies, or controls are synthesized or refined using principled, modular procedures rather than monolithic function approximators. PIBR subsumes a wide spectrum of settings: classical Bayesian games, polyhedral strategy spaces, nonlinear trajectory games, and, more recently, code-based policies for Markov games via LLMs. Its central object is the iterated application of a best-response operator, instantiated via diverse mechanisms depending on the domain and representation.

1. Core Framework and Motivations

At its core, PIBR proceeds by alternating optimization: each player, holding others’ strategies fixed, computes its best response. The motivation traces back to classical best-response dynamics for Nash computation, piecewise-linear methods for Bayesian games, and sequential optimization for continuous trajectories. The programmatic viewpoint further allows agents to introspect or directly condition on their opponents’ strategy representations—reaching beyond black-box (opaque) policy regimes—thus operationalizing learning in the space of symbolic (often interpretable) policies and facilitating nuanced equilibrium concepts such as Program Equilibrium (Lin et al., 24 Dec 2025, Reeves et al., 2012).

In modern instantiations, PIBR addresses two major representational bottlenecks:

• Neural policies’ incomprehensibility for direct conditioning or code-based reasoning.
• Intractability of direct optimal-response computation in complex or high-dimensional settings.

PIBR addresses these via explicit, modular subroutines (e.g., LLM-based interpreters, best-response oracles, convex relaxations), tractably iterating optimization in restricted policy or strategy spaces (Lin et al., 24 Dec 2025, Chakrabarti et al., 2023, Reeves et al., 2012, Sin et al., 2020).

2. Algorithmic Realizations by Domain

PIBR algorithms are concretely implemented via mechanisms tailored to the game model and policy representation:

a) Code-based Markov Games with LLM Synthesis

Policies are represented as Turing-complete program source code, with execution and best-response synthesis delegated to LLMs. Given an environment description and an opponent’s source code, an LLM acts as a point-wise best-response operator: it interprets the opponent, generates candidate best-response code, and accepts structured optimization feedback (utility and unit-test losses). Textual gradient methods (TextGrad) differentiate through the LLM generation process to refine the prompt and optimize performance (Lin et al., 24 Dec 2025).

Program Equilibrium

Agents may introspect or condition on the full code of their counterparts, operationalizing program equilibria as in Tennenholtz (2004). This approach yields both transparent policy adaptation and the possibility of direct code-level cooperation or strategic matching.

b) Polyhedral Games via Best-Response Oracles

For games with polyhedral (e.g., sequence-form or simplex) strategy sets, PIBR leverages linear minimization oracles (LMOs) as best-response engines. The outer algorithm proceeds via an iterative saddle-point or OMD/FW loop, where each proximal update is approximately solved using Away-Step Frank-Wolfe subroutines and each subproblem calls only the best-response oracle a logarithmic number of times (Chakrabarti et al., 2023).

Theoretical Guarantees

This structure ensures constant regret in zero-sum, $O(T^{1/4})$ regret in general-sum, linear-rate last-iterate convergence in zero-sum under metric subregularity, and best-iterate convergence of $O(1/\sqrt{T})$ without such assumptions.

c) Piecewise Linear Bayesian Games

In infinite games of incomplete information, player strategies are represented as piecewise linear functions; each best-response is computed via analytic maximization of expected utility, yielding again a piecewise linear function. Iteration yields Bayes-Nash equilibria, with efficient convergence in typical auction and bargaining games (Reeves et al., 2012).

d) Optimal Control in Multi-Body Trajectory Games

For multi-body pursuit-evasion scenarios, each agent’s control trajectory is optimized with fixed opponent trajectories, using Sequential Convex Programming as a subroutine at each PIBR step. The PIBR loop alternates the solution of convexified OCPs for pursuers and evaders, iterating until convergence in trajectory space (Sin et al., 2020).

3. Detailed Workflow: The PIBR Loop

A canonical PIBR algorithm proceeds as follows:

1. Initialize strategies/policies: Start from program stubs, piecewise-linear forms, or feasible points in the strategy space.
2. Iterative optimization: At each outer iteration, select an agent and fix the current opponents’ strategies.
3. Inner best-response computation: For the selected agent, synthesize or optimize a (programmatic) best response to the opponents’ strategies. The mechanism depends on the domain:
   • LLM-based code generation and optimization for programmatic policies.
   • Frank-Wolfe oracles for polyhedral games.
   • Symbolic maximization for piecewise linear response in Bayesian games.
   • Trajectory optimization via SCP for nonlinear control problems.
4. Utility and feasibility evaluation: Score the candidate response by expected return, unit test/syntactic validity, or constraint satisfaction.
5. Policy update: Adopt the best candidate from the inner loop.
6. Convergence and return: Loop over agents; track convergence via utility, social welfare, or sup-norm difference between strategies. Output the equilibrium or best-performing profile.

4. Experimental Protocols and Empirical Results

Fields of application and experimental protocols vary across PIBR instantiations:

Domain	Policy/Strategy Representation	Inner Oracle	Main Metrics	Reported Results
Markov/Coordination	Program code (Python)	LLM + TextGrad	Social welfare, pass rate	Instantaneous convergence to optimal welfare; robust synthesis of cooperating code (Lin et al., 24 Dec 2025)
Polyhedral games	Sequence/simplex vectors	LMO, FW	Regret, duality gap	$O(\log t)$ oracle calls/iteration, $O(1/T)$ to Nash, $O(T^{1/4})$ in general-sum (Chakrabarti et al., 2023)
Bayesian auctions	Piecewise-linear functions	Symbolic maxim.	Uniform error, #segments	Analytical best-responses, rapid convergence (1–10 steps), polynomial runtime (Reeves et al., 2012)
Asset-guarding OCP	State/control trajectories	Convex QP/SOCP	Trajectory change, cost	Convergence in 12–19 PIBR rounds, empirical local equilibrium (Sin et al., 2020)

In all domains, compared to uninformed baselines (e.g., random code mutation, zero-shot LLM prompting), PIBR achieves faster convergence, higher robustness, and (for code-based settings) improved code validity and interpretability. In Markov games, ablation studies reveal critical dependency on jointly optimizing for both code correctness (unit test loss) and utility (return), with omission of either loss yielding collapse to invalid or trivial policies (Lin et al., 24 Dec 2025).

5. Program Representations, Introspection, and Policy Conditioning

The “programmatic” aspect differentiates PIBR from conventional best-response dynamics by explicitly representing and manipulating strategies:

• Source code policies: Agents can condition on the actual source code of their opponents, inspect logic, or synthesize strategies contingent on the opponent’s decision procedure—not merely on observed trajectories. This enables realization of program equilibrium, where introspection supports richer forms of cooperation or punishment (Lin et al., 24 Dec 2025).
• Symbolic or modular forms: In Bayesian and polyhedral settings, piecewise-linear or modular decompositions permit tractable, analytic best-response synthesis and explicit verification of policy structure (Reeves et al., 2012, Chakrabarti et al., 2023).

This shift also introduces new operational bottlenecks, mainly in code synthesis (LLM runtime cost), code validation (need for robust unit testing), and scalability (automatic generation of tests or automated type checking).

6. Theoretical Guarantees and Limitations

PIBR’s theoretical properties diverge based on domain and oracle assumptions:

• Zero-sum polyhedral games: Guarantees include constant or vanishing regret, linear-rate last-iterate convergence under subregularity, and efficient use of best-response oracles; facial distance quantifies conditioning (Chakrabarti et al., 2023).
• Piecewise-linear Bayesian games: Existence of pure-strategy equilibria and rapid convergence hold under compactness, continuity, and single-crossing properties. Cyclic behaviors may arise in exceptional asymmetric games (Reeves et al., 2012).
• Code-based Markov games: There is no general convergence proof; in practice, convergence may be fast for simple environments but exhibits variance and local optima in high-dimensional or rich coordination games (Lin et al., 24 Dec 2025).
• Nonlinear dynamic games: Empirical convergence is observed, but global equilibrium guarantees are absent. SCP ensures feasible trajectory tracking but is subject to local minima (Sin et al., 2020).

The robustness and convergence of PIBR with LLM-based coders fundamentally depend on the expressiveness of the code space, the coverage of unit tests, and the ability of LLMs to understand and synthesize correct logic. Removing key loss components (utility or unit-test loss) degrades performance, causing the algorithm to gravitate to non-strategic or invalid code (Lin et al., 24 Dec 2025).

7. Future Directions and Open Problems

PIBR remains a rapidly evolving area with several open avenues:

• Extension to N-agent settings and complex interaction graphs.
• Integration of automated test synthesis or formal program-analysis techniques for programmatic policies, enabling greater robustness and coverage in policy correctness validation (Lin et al., 24 Dec 2025).
• Hybrid architectures: Combining neural policies (for continuous state/action or low-level control) with programmatic policies (for strategic reasoning or high-level planning).
• Unifying framework and theory: Sharpening convergence guarantees for empirical, LLM-based PIBR, including under approximate best responses and non-stationary optimization.
• Scalability and efficiency: Mitigating computational cost, especially in LLM-in-the-loop systems, via smarter caching, reduced calls, or curriculum-based training (Lin et al., 24 Dec 2025).

Empirical findings in both structured and programmatic environments reinforce the versatility of PIBR as a unifying operational paradigm to tackle equilibrium computation, interpretable policy synthesis, and introspective strategy adaptation, leveraging advances in program synthesis, language modeling, and convex optimization (Lin et al., 24 Dec 2025, Chakrabarti et al., 2023, Reeves et al., 2012, Sin et al., 2020).