Convergence to collusion in algorithmic pricing

Abstract: Artificial intelligence algorithms are increasingly used by firms to set prices. Previous research shows that they can exhibit collusive behaviour, but how quickly they can do so has so far remained an open question. I show that a modern deep reinforcement learning model deployed to price goods in a repeated oligopolistic competition game with continuous prices converges to a collusive outcome in an amount of time that matches empirical observations, under reasonable assumptions on the length of a time step. This model shows cooperative behaviour supported by reward-punishment schemes that discourage deviations.

• The paper demonstrates that the soft actor-critic deep RL approach reduces collusion convergence time by up to two orders of magnitude compared to Q-learning.
• It employs continuous-space, policy gradient methods in a repeated Bertrand model to capture the rapid emergence of supracompetitive pricing and robust punishment strategies.
• Empirical results indicate that algorithmic agents quickly settle into collusive equilibria, raising significant antitrust concerns in modern oligopolistic markets.

Convergence to Collusion in Algorithmic Pricing: Deep RL and Supracompetitive Outcomes

Introduction

This work presents a thorough investigation of the emergence and timescale of collusive pricing in oligopolistic markets where firms deploy reinforcement learning (RL)-based algorithms. Existing literature established that even model-free Q-learning agents can sustain supracompetitive pricing, but with convergence timescales that are implausible for most real-world applications—often requiring many hundreds of thousands of games. This paper addresses that open question directly by leveraging advanced actor-critic RL architectures to both accelerate and systematize the convergence to collusive equilibria.

Background and Problem Formulation

The canonical model considered is repeated Bertrand competition with continuous pricing, differentiating products under logit demand and constant marginal costs. Unlike prior studies relying on Q-learning—which discretizes the action space and suffers from slow convergence, high sample complexity, and limited generalization—this work adopts continuous-space, function approximation-based policy gradient methods, specifically the average-reward soft actor-critic (SAC) algorithm.

Actor-critic methods, by maintaining parametric policy and value function approximators and conducting gradient-based optimization, can exploit the topological structure of state and action spaces for efficient policy discovery, naturally support continuous actions, and regularize the exploration process. This is essential for moving closer to economic realism in modeling pricing algorithms.

Algorithmic Innovations

The transition to SAC introduces several distinguishing features. First, learning is performed over continuous state and action spaces, avoiding information loss induced by discretization. Second, the soft actor-critic framework does not maximize discounted return (which is ill-posed with function approximation over continuous state space) but directly optimizes the long-run average profit ($\bar{\pi}$) via differential value functions. This provides both theoretical tractability and practical relevance.

SAC achieves adaptive, state-conditional exploration by regularizing the policy's entropy with a KL-divergence penalty, controlled through a dynamically-adjusted temperature parameter. This stochasticity is fundamental to both supporting exploration and maintaining agents' ability to punish deviations.

Empirical Results

Speed and Nature of Convergence

The empirical section quantifies the profit gain dynamics as soft actor-critic policy learning unfolds. The principal result is a two-orders-of-magnitude reduction in the time required to reach supracompetitive pricing relative to Q-learning. Collusive policy profiles typically emerge and stabilize within 50,000 periods, which, when mapped to an hourly price adjustment schedule, is on the order of five years—consistent with real-world transitions observed in empirical studies such as algorithmic adoption in the German gasoline market.

Figure 1: Training curves demonstrate that average profit gain increases from competitive to collusive levels, converging rapidly across runs.

The distribution of profit gains across random seeds is broad, with most runs achieving near-uniform pricing benchmarks (approximately 40% of monopoly gain), and a substantial right tail exceeding 50%.

Figure 2: The histogram of profit gains indicates high variance; a non-negligible fraction of experiments sustain profits much closer to the monopoly benchmark.

Collusive Equilibrium and Punishment Strategies

A central theoretical challenge is distinguishing superficial price elevation (myopic high pricing) from genuine collusive strategies that are robust to deviation. The paper shows that agents learn non-trivial reward-punishment responses: after a best-response deviation, the deviating agent is met with an immediate one-period punishment, after which prices gradually recover toward the previous collusive fixed point following a tit-for-tat scheme.

Figure 3: Average post-deviation price trajectories reveal immediate punishment and gradual recovery, characteristic of contingent collusive strategies.

Impulse response distributions across runs demonstrate that this behavior holds systematically:

Figure 4: Boxplots confirm that the reduction and subsequent recovery in prices after a deviation are robust features of Nash-convergent runs.

Moreover, learned policy phase diagrams visualize the system dynamics:

Figure 5: Phase diagrams show strategy profile fixed points, with convergence toward supracompetitive equilibrium and robust return trajectories after deviations.

Sensitivity to Market Complexity

When the number of competing agents is increased to three, mean profit gains decrease (~25% relative to monopoly), impulse response variance rises, and the fraction of runs reaching equilibrium shrinks (about 30%). Collusion becomes progressively more difficult with market fragmentation, but convergence timescales remain similar.

Figure 6: In three-agent settings, average profit gains remain significant but are reduced and more variable than in duopoly.

Figure 7: The impulse response to deviation in the three-agent case exhibits higher variance and a less consistent punishment pattern.

Robustness and Generalization

Across hyperparameter sweeps and alternative architectures, reward-punishment collusive strategies persist, though success rates (i.e., convergence to Nash equilibrium) are highly contingent on hyperparameter selection and initialization—a familiar outcome in deep RL.

Theoretical and Practical Implications

This research demonstrates that the adoption of modern deep RL architectures by competing firms is sufficient to support rapid endogenous convergence to collusive pricing—without requiring explicit encoding of economic theory or inter-agent communication. The findings reinforce antitrust concerns: even in the absence of model-based coordination, function-approximation-based RL can drive markets toward supracompetitive outcomes on commercially relevant timescales.

From a theoretical standpoint, the nature of punishment strategies (contingent, not grim-trigger; robust to local deviations but not globally optimal) highlights the role of memory constraints and functional representability in shaping strategic equilibrium selection. The results also caution against overreliance on discounted reward formulations in continuous control settings—average reward frameworks prove more appropriate, yet introduce nontrivial selection effects and convergence pathologies.

The analysis also invites a reevaluation of notions such as "offline learning" and algorithmic rematching in business practice, given that online learning with modern RL methods appears sufficient for sustained collusion.

Future Directions

The study opens numerous avenues for further exploration. Rigorous characterization of the full equilibrium correspondence for deep RL-trained strategies, in analogy to the folk theorem, remains open. Extending to more complex environments (asynchronous or sequential moves, entry/exit dynamics, partial observability), and further empirical validation against field data are also essential steps. An investigation into the robustness of learned collusive equilibria to exogenous shocks and policy interventions would be directly relevant for regulators. There is also a need to analyze the interplay between market structure and algorithm choice in shaping collusion feasibility.

Conclusion

The paper establishes that function approximation-based deep RL—specifically the soft actor-critic algorithm—enables pricing agents in oligopolistic settings to converge to supracompetitive, collusive strategies with realistic convergence timescales and robust punishment mechanisms. This paradigm shift in sample efficiency and strategic sophistication relative to Q-learning compels both the economic theory and antitrust policy literatures to account for rapid, decentralized algorithmic collusion in modern markets. Future work should elucidate the structural, computational, and policy determinants of equilibrium selection in algorithmic competition.