CalConflictBench: Adaptive Calendar Conflict Benchmark

• CalConflictBench is a benchmark that evaluates language models on long-horizon, sequential calendar conflict resolution with user-adaptive preference inference.
• It simulates realistic organizational scheduling conflicts using synthetic, chronologically-ordered event streams across diverse user roles.
• The benchmark incorporates the PEARL framework, leveraging external memory and curriculum-shaped rewards for progressive adaptation and improved decision accuracy.

CalConflictBench is a large-scale benchmark designed for the evaluation of language agents and LLMs on long-horizon, sequential calendar conflict resolution with user-adaptive preference inference. The benchmark presents synthetic, chronologically-ordered streams of overlapping meeting and event invitations, modeled to reflect realistic organizational settings, and requires agents to make contextually consistent, personalized scheduling decisions. CalConflictBench is accompanied by PEARL, a reinforcement learning framework that combines external memory in the form of a "Strategy Hub" and curriculum-shaped, round-wise reward optimization. This framework facilitates progressive adaptation to user preferences over time, a critical capability for automating complex scheduling logistics that are otherwise labor-intensive and error-prone (Li et al., 17 Jan 2026).

1. Dataset Generation and Benchmark Structure

CalConflictBench comprises four synthetic organizations, each with ten users assigned roles such as PI, postdoc, PhD student, CEO, engineer, and HR. Each user receives a year-long calendar auto-generated from role-specific schemas characterizing event recurrence, topics, attendee sets, and durations. Two conflict rounds are injected weekly—104 rounds per user—where one "anchor" event is deliberately overlapped with one to four competing events (with the main evaluation at M=5 events per conflict). All events in a conflict set occur in the same time slot (±5 minutes), and attribute diversity is ensured through structured "conflict reason" operators (urgency, topic, modality).

User preferences are parametrized through a set of "priority principles" for each role: $P(r) = \{ \langle p_k, w_k, g_k(\cdot) \rangle \}$, where each principle $p_k$ is triggered via a context-sensitive function $g_k$ and weighted by $w_k$. The ground-truth decision for each conflict maximizes a principle-weighted score across all candidate events.

Human verification employs a three-stage trust pipeline: annotators assess event realism, organizational coherence, and ground-truth alignment with preference principles for all labeled acceptances.

Conflict rounds are presented sequentially. At round $t$, the agent observes:

• The current calendar $C_t$
• Contextual organization and role metadata
• The conflict set $E_t = \{e_t^1, \ldots, e_t^M\}$

Agent outputs per round include: the accepted event, declined events, a full candidate ranking, and a natural-language rationale. Immediate feedback is provided, but the underlying principle set $P(r)$ is never directly revealed.

2. Problem Formalization and Evaluation Metrics

The benchmark is formulated as a deterministic Markov Decision Process (MDP):

• State: $S_t = (C_t, E_t, \text{context info})$
• Action: $$A_t = \{ a_t^i \in \{0,1\}_{i=1..M} \mid \sum_i a_t^i = 1 \}$$ (single accept decision)

The calendar is updated deterministically: $p_k$0.

Metrics are evaluated both per round and instance-level:

• Decision accuracy: $p_k$1 if the accepted event matches ground-truth; $p_k$2 otherwise.
• Optimal Rank Distance (ORD): $p_k$3, where $p_k$4 is the agent's ranking and $p_k$5 the correct accept.
• Average Error Rate (AER): $p_k$6 over $p_k$7 rounds.
• Error Reduction Rate (ERR): $p_k$8, comparing first and last quarter error rates.

The evaluation protocol enforces a single-turn interface per round, with access to up to $p_k$9 rounds of calendar history and the same prompt template for all agents.

3. Baseline Agents and Performance

CalConflictBench evaluates a range of baseline LLMs (Qwen3-4B/8B/14B/30B, LLaMA-3.1-8B, GPT-5, OLMo3, Gemini-2.5-flash) and agentic variants (ReAct, Mem+ReAct). Chain-of-thought and memory-augmented settings are also tested.

Key observed metrics for $g_k$0 rounds, $g_k$1, $g_k$2 (over 10 users):

• Qwen3-30B-Think: AER=0.35, ERR=0.161
• Qwen3-8B: AER=0.37, ERR=0.026
• GPT-5: AER=0.35, ERR=0.092
• ReAct: AER=0.39, ERR=0.007
• Mem+ReAct: AER=0.40, ERR=–0.162

The strongest single-turn supervised fine-tuned baseline achieves AER=0.27, ERR=0.325. None of the non-PEARL agents exhibit significant preference adaptation or error reduction over the time horizon, indicating an inability to progressively model user-specific priorities (Li et al., 17 Jan 2026).

4. The PEARL Framework: Architecture and Learning Procedure

PEARL introduces two core innovations:

1. External Memory: "Strategy Hub" — a bounded (max 10) natural-language list encoding inferred user preference heuristics (e.g., “Always attend deadline-driven calls”). The hub exposes two tool-API actions: list() to retrieve all strategies and update(Δ) to add or revise a strategy, with persistence across rounds.
2. Curriculum-Shaped Round-Wise Reward — a composite reward signal for each round combines validity, accuracy, ranking performance, and hub usage with a time-dependent weighting schedule:

   $g_k$3

   with $g_k$4 and $g_k$5, $g_k$6.

Structured rollouts proceed as follows: for each round, an agent may interact internally up to $g_k$7 times, choosing either a hub action (memory update) or a decision/rationale action, after which the environment returns feedback based on the above reward scheme. Per-round advantages are computed using group mean and standard deviation normalization, and gradient optimization is performed using clipped PPO (GRPO) with KL regularization coefficient 0.001.

5. Empirical Results and Ablation Analysis

PEARL achieves a substantial reduction in final-round average error rate: from 0.27 (best SFT) down to 0.12 (55% improvement). The error reduction rate (ERR) increases from 0.325 to 0.761, establishing that the agent not only succeeds in immediate decision accuracy but also adapts its behavior to user preferences over time.

Ablation studies demonstrate that "zero-shot+hub" alone (AER=0.41, ERR=0.048) is insufficient; both the reward curriculum and persistent memory are necessary for pronounced long-horizon adaptation. Early rounds of PEARL training emphasize memory formation through the strategy hub (high $g_k$8), while later rounds prioritize ranking correctness (high $g_k$9), producing robust temporal adaptation.

6. Limitations, Failure Modes, and Prospective Directions

Baseline LLMs exhibit flat or negative ERR, with larger context windows or agentic prompting (ReAct, Mem+ReAct) failing to deliver consistent adaptation. This suggests intrinsic limitations in LLMs' capacity for on-the-fly, sequential preference learning without explicit externalized memory or reward shaping.

CalConflictBench and the PEARL framework illuminate gaps in current LLM scheduling agents, particularly for scenarios requiring dynamic, feedback-driven user modeling. A plausible implication is that scalable, trustworthy automation of calendaring depends on both persistent preference memory and temporally aware optimization objectives.

Potential future research avenues include:

• Extending the dataset to real-world calendars with richer, non-synthetic preference signals.
• Exploring alternative memory architectures or tool APIs for preference tracking.
• Generalizing the round-wise curriculum concept to other sequential decision-making domains with evolving user intent.

CalConflictBench thus establishes a rigorous foundation for benchmarking language agent performance in long-term, adaptive conflict resolution, supporting detailed analysis of preference inference and adaptation dynamics (Li et al., 17 Jan 2026).