Learning to Bid in Repeated Second-Price Auctions with Dynamic Values and Aggregated Feedback

Abstract: We study the problem of learning to bid when the bidder's value is dynamic, i.e., when the current value depends on past outcomes. Specifically, we consider a bidder participating in repeated second-price auctions whose value depends on the time elapsed since their last successful bid, with auctions arriving in continuous time and only aggregated feedback revealed at the end of the horizon. Such a bidder must (1) balance the immediate benefit of winning the current auction against its impact on future values and (2) learn unknown environmental parameters. We derive regret bounds for a class of learning methods that combine plug-in estimators with a differential-equation characterization of the optimal policy, and show that a specific confidence bound algorithm learns the optimal policy with a near optimal regret of $\widetilde{O}(\log N)$ for piecewise linear primitives, and $\widetilde{O}(N^{1/3})$ for general, smooth primitives, achieving these regrets without explicit randomization. These theoretical results are supported by numerical experiments.

• The paper introduces a dynamic programming framework that uses ODE-based control to derive an optimal bid policy under dynamically changing values.
• It integrates plug-in estimation of key functions with various algorithms, achieving competitive regret bounds including logarithmic and sublinear rates.
• The study validates practical applicability for ad bidding systems and opens avenues for extensions to other auction formats and resource allocation scenarios.

Learning to Bid in Repeated Second-Price Auctions with Dynamic Values and Aggregated Feedback

Problem Setting and Motivation

The paper "Learning to Bid in Repeated Second-Price Auctions with Dynamic Values and Aggregated Feedback" (2605.28133) addresses optimal and efficient learning strategies for a recurrent bidder in repeated second-price auctions where the bidder’s value is dynamic—specifically, when the current value depends on the elapsed time since the last win. This setting is motivated by applications in digital advertising, where user fatigue and recency effects cause a bidder’s marginal value to vary with exposure history. Unlike the static value setting, bidding the instantaneous true value is suboptimal under such dynamics, invalidating standard truthful bidding arguments.

A critical further constraint is that the bidder observes only aggregated feedback: at the end of each episode (sequence of auctions), only the sum of realized values is observed, not individual auction outcomes. The auction arrival process and episode length are stochastic (Poisson processes), the competitive landscape is stationary, and both the reward function $k$ (mapping time-since-last-win to value) and the competition distribution $q$ are unknown and must be learned.

Theoretical Contributions

Dynamic Programming and Control Formulation

Building on a continuous-time, dynamic-programming-based control formalism, the paper leverages ordinary differential equation (ODE) characterizations for the value and optimal bid policy. The Bellman value $V^\star(\tau)$ at recency $\tau$ solves an ODE with a nonstandard reset condition upon a win. The optimal bidding policy $\pi^\star(t)$ admits a closed-form characterization in terms of $k$, $q$, and $V^\star$.

The paper's core technical contribution is the integration of plug-in estimators for the unknown primitives $k$ and $q$ with this ODE-based solver. The central result (the "Extension Invariance" Theorem) proves that only estimation of $q$0 up to the maximal bid $q$1 exercised by the current policy is required for optimality, decoupling the learning region from the entire domain and connecting policy structure to sample complexity.

Learning Algorithms and Regret Bounds

The regret framework is formalized as cumulative expected value loss relative to the true optimal policy over $q$2 episodes. Four principal algorithms are analyzed:

1. Greedy Plug-in (Asymptotic Consistency): Iteratively estimate $q$3, $q$4 from collected data, solve for $q$5, rollout, and repeat. No explicit exploration or randomization is required. This procedure converges to optimality as long as the sequence of policies eventually visits bids covering the region of the true optimal bid. However, there are no finite-time regret guarantees for arbitrary $q$6.
2. Two-Phase (Explore-then-Commit):
   • Phase 1: Use a fixed exploratory policy (e.g., always bid high) for $q$7 episodes.
   • Phase 2: Commit to a plug-in policy for the remaining $q$8 episodes.
   • Yields high-probability, $q$9 regret.
3. Three-Phase (Decoupled Estimation):
   • Phase 1: Estimate $V^\star(\tau)$0 with minimal bidding over $V^\star(\tau)$1 episodes.
   • Phase 2: Estimate $V^\star(\tau)$2 using a policy derived from the $V^\star(\tau)$3 estimate for $V^\star(\tau)$4 episodes.
   • Exploitation in the remainder.
   • Also achieves $V^\star(\tau)$5 regret.
4. Confidence Bounds/UCB-Type Algorithm:
   • Construct upper/lower confidence envelopes for $V^\star(\tau)$6/$V^\star(\tau)$7 (respectively).
   • The policy is optimized for the most optimistic plausible $V^\star(\tau)$8 and most pessimistic plausible $V^\star(\tau)$9.
   • Achieves time-uniform high-probability regret $\tau$0 for piecewise linear $\tau$1, $\tau$2.
   • For general smooth primitives, regret is $\tau$3.

Critically, none of these algorithms require explicit randomization for exploration, a consequence of the extension invariance property and structural monotonicity results regarding the effect of $\tau$4 or $\tau$5 perturbations on the maximal optimal bid.

Lower Bound

The paper derives a logarithmic lower bound: any algorithm suffers regret at least $\tau$6 under mild regularity, even with aggregated feedback. The UCB-style procedure is thus rate-optimal.

Identification and Estimation Procedures

• Value Estimation ($\tau$7): Modeled as a monotone, concave, piecewise linear function fitted by projected OLS. Aggregated feedback (per episode) allows casting the estimation as a standard regression, with theoretical guarantees under mild coverage conditions on the exploratory policy.
• Competition CDF Estimation ($\tau$8): Also piecewise linear with nonnegative slopes, estimated by MLE leveraging observed win/loss and, when won, the observed price. Estimation error on $\tau$9 is shown to be $\pi^\star(t)$0, assuming sufficient coverage of high bids.

Numerical Results

Empirical evaluation corroborates theoretical guarantees. In environments with $\pi^\star(t)$1, $\pi^\star(t)$2, the proposed UCB-style algorithm matches the predicted $\pi^\star(t)$3 rate for piecewise linear absorption and sublinear rates for smooth $\pi^\star(t)$4, $\pi^\star(t)$5.

Implications and Future Directions

The paper makes a strong technical advance by closing the regret-optimality gap for this class of recurrent, dynamic-value auction problems under extremely weak feedback. It demonstrates that model-based RL techniques (with function estimation and continuous-time control) can be adapted for practical, guaranteed online learning in auction environments where recency effects dominate.

From an applied perspective, the results have direct implications for ad bidding systems and repeated resource allocation problems affected by user fatigue, exposure capping, or contract fulfillment. The algorithmic analysis implies that practical, scalable solvers are possible without hand-crafted exploration heuristics.

Theoretically, the work underlines the importance of problem geometry—especially extension invariance and state reset—in determining identifiability and learnability. The ODE-based analysis and sample complexity results potentially inform a broader class of reinforcement learning/control problems where dynamics exhibit resets or where learning a restricted parametric class is sufficient for optimality.

Open directions include:

• Extending the analysis to non-second-price mechanisms (first-price, GSP) where the lack of monotonicity and different payment rules effect incentive compatibility and identifiability.
• Relaxing episodic aggregation to streaming or delayed attribution settings.
• Generalizing the dynamic value structure to dependence on the full bid-win-loss history, stochasticity, and state augmentation for heterogeneity (e.g., per-item user effects).
• Applying these techniques in finance, supply chain, and resource allocation settings with resetting constraints and partial feedback.

Conclusion

This work provides a comprehensive and rigorous analysis of learning to bid in repeated second-price auctions with dynamic bidder values and aggregated feedback. It synthesizes continuous-time control, function approximation, and nonasymptotic online learning, supplying both optimal regret rates and practical, implementable algorithms. Extensions to broader recurrent and strategic learning settings appear viable and merit further investigation.