Curriculum-Guided Competitive Training

• Curriculum-Guided Competitive Training is a framework that integrates automated curriculum generation with competitive scenarios to enhance both agent and human learning.
• It employs techniques such as adversarial self-play, entropy regularization, and dynamic regret updates to adapt challenges based on learner performance.
• Practical implementations in reinforcement learning and contest-based education demonstrate improved generalization, robust skill acquisition, and real-world applicability.

Curriculum-Guided Competitive Training refers to strategic frameworks that combine automated, adaptive curriculum generation with structured competitive scenarios to optimize agent or human learning under real-world constraints. This approach targets both the progressive acquisition of skills and robust generalization, utilizing mechanisms from multi-agent reinforcement learning (RL) and contest-based pedagogy. In artificial agent settings, it leverages adversarial and cooperative dynamics to shape learning trajectories, while in human education, it instantiates authentic competition-driven assessments within an interleaved curriculum structure. Core techniques include regret-based self-play, entropy-regularized diversification, dynamic goal updating, and continuous formative assessment.

1. Formal Frameworks: Symmetric Multi-Agent Curriculum Self-Play

Automated curriculum generation in RL is operationalized via multi-agent adversarial games, most notably in the Curriculum Self-Play (CuSP) model. This framework is grounded in a goal-conditioned Markov Decision Process (MDP), $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, r, \mathcal{G}, \gamma)$$, where policies must solve an adaptive set of target goals $g \in \mathcal{G}$ under changing environmental and adversarial conditions.

The CuSP structure involves four agents—two students (policy learners, $\pi_A$ and $\pi_B$) and two teachers (goal generators, $G_A$ and $G_B$). Teachers compete and cooperate via regret maximization: $$\mathfrak{R}^{G_A}(g) = R^A(g) - R^B(g), \qquad \mathfrak{R}^{G_B}(g) = R^B(g) - R^A(g)$$ where $R^i(g)$ is the $\gamma$-discounted return for student $i$ on goal $g \in \mathcal{G}$0. This structure generalizes and symmetrizes the PAIRED game, yielding a two-team, zero-sum game over goal proposals and policy responses. Each student seeks to maximize their expected return: $g \in \mathcal{G}$1 This fully symmetric setup enables robust automatic curriculum induction, overcoming stagnation arising from student divergence in earlier, asymmetric approaches (Du et al., 2022).

2. Entropic Goal Coverage and Dynamic Regret Updates

Efficient curriculum generation requires maintaining diversity and relevance in goal proposals. CuSP introduces entropic exploration into goal-generator training, employing a maximum-entropy RL objective: $g \in \mathcal{G}$2 where $g \in \mathcal{G}$3 weights entropic coverage to mitigate mode collapse and encourage broad exploration.

Because students are non-stationary, previous goals’ regret profiles become outdated as learning and forgetting occur. CuSP employs dynamic regret updating: for each stored goal-regret pair $g \in \mathcal{G}$4, updates are performed via the students’ critics: $g \in \mathcal{G}$5 with $g \in \mathcal{G}$6 modulating memory. This ensures that goal prioritization continually adapts to current student capabilities and weaknesses (Du et al., 2022).

3. Algorithmic Implementation and Training Workflow

The CuSP training algorithm consists of iterative rounds comprising:

• Dual goal sampling ($g \in \mathcal{G}$7, $g \in \mathcal{G}$8).
• Rollouts: In the “easy” setting, students attempt the goals proposed by their “friendly” teachers; in the “hard” setting, each student attempts the other’s assigned goal.
• Reward and regret calculation, storage in teacher replay buffers, and dynamic regret updates via student critics.
• Off-policy SAC updates for both student policies and teacher goal-generators, with students’ experiences stored in a shared buffer and teachers’ replay buffers exceeding $g \in \mathcal{G}$9 entries to ensure diverse coverage.
• Hyperparameter recommendations include learning rates $\pi_A$0, entropy coefficient $\pi_A$1, batch size 1024, and dynamic regret update intervals calibrated per environment.

Teacher networks’ outputs are bounded by Tanh scaling to match $\pi_A$2, and symmetrization is achieved by proposing dual goals per round. Continuous regret refresh at each round is crucial for adapting to non-stationary learners. Off-policy SAC training for teachers is essential to overcome flat regret landscapes during early training, outperforming on-policy alternatives (Du et al., 2022).

4. Applications in Human Learning: Contest-Based Competitive Programming Curricula

In the field of human learning, Curriculum-Guided Competitive Training manifests in structured, contest-based courses for competitive programming. At Purdue University, the CP3 course exemplifies this approach by embedding weekly ICPC-style contest sessions, requiring students to solve algorithmic problems under real-world time pressure. The pedagogical triad targets:

• EO-1 (Observation): mapping novel problems to familiar templates.
• EO-2 (Technique): mastery of discrete algorithms and data structure toolkits.
• EO-3 (Implementation): efficient, error-free coding and debugging.

Assessment is grounded in a scoring formula: $\pi_A$3 where $\pi_A$4 is the in-class problem count (2 points each) and $\pi_A$5 the number of upsolved post-contest problems (1 point each). The aggregate over sessions, with drop-of-lowest policies, provides continuous feedback and drives iterative improvement (Luo, 1 Apr 2025).

Contest scheduling interleaves individual and team events, exposing students to realistic collaboration and leader selection protocols, while formative assessment artifacts (reflections, code presentations, Google Sheets tracking) close the feedback loop. The curriculum sequence balances technique-heavy and observation-heavy weeks for staged skills development.

5. Empirical Performance and Generalization

CuSP’s empirical evaluation covers continuous-control RL tasks (e.g., Point-Mass Obstacle, Walker navigation, Reach, Toss, Pick-and-Reach), using both sparse and dense rewards. The framework’s automatic curricula enabled RL agents to achieve higher success rates on out-of-distribution goal sets, $\pi_A$6, compared to baselines such as Domain Randomization, GoalGAN, and ASP+BC. Notable improvements occur for tasks with asymmetric difficulty landscapes.

Additional findings include robustness to misspecified or infeasible goals: when goal spaces are artificially expanded, CuSP maintains high success rates, whereas alternative methods degrade. Emergent skills—efficient maze navigation, rapid locomotion, and complex manipulation—arise as a byproduct of curriculum diversity and regret balancing (Du et al., 2022).

In competitive programming education, embedding authentic contests and cooperative feedback accelerates learner progression, as evidenced by student grade distributions and external contest outcomes (ICPC World Finals and North America Championships). While long-term rating improvements remain under study, initial application of contest-based curricula yields outcomes consistent with rapid skill development (Luo, 1 Apr 2025).

6. Theoretical Foundations and Design Considerations

The formal underpinning of curriculum-guided competitive training in RL resides in zero-sum games between agent-policy and goal-generator pairs. Finite policy and goal spaces guarantee Nash equilibria by Nash’s theorem; more broadly, compactness and continuity conditions invoke Glicksberg’s theorem, though neural approximation introduces practical limitations.

Implementation details critical for empirical success include large, entropy-regularized teacher replay buffers; per-round regret refreshing; and off-policy training algorithms. Design elements such as team composition, problem temporization, and scaffolded reflection activities further enhance the transfer of competitive training methodologies to human education settings (Du et al., 2022, Luo, 1 Apr 2025).

7. Future Directions and Implications

Current approaches optimize both progressive feasibility and long-horizon generalization, with mechanisms to mitigate catastrophic forgetting and mode collapse. Open questions remain regarding the scaling of curriculum-guided competitive training to high-dimensional, real-world domains, the integration of explicit difficulty calibration, and longitudinal tracking of skill transfer—especially in human education.

A plausible implication is that further advances in automatic curriculum generation and contest-based formative assessment are likely to converge in more generalized educational and autonomous agent systems, blending regret-based optimization with authentic competitive scenarios for accelerated, reliable skill acquisition.