Semi-Online Performance (SOP) Metric

• SOP Metric is defined as a performance measure that interpolates between pure offline optimality and worst-case online bounds using competitive ratios and success rates.
• It utilizes methodologies such as multi-turn rollouts with patching, regression of offline proxies to online KPIs, and structured decision graph traversal to capture practical performance.
• SOP metrics have significant applications in matching theory, scheduling, reinforcement learning, LLM agentic frameworks, and recommender systems, providing robust evaluation under uncertainty.

The Semi-Online Performance (SOP) metric denotes a class of evaluation frameworks and formulas designed to measure algorithmic or system efficacy in environments that combine aspects of online and offline information or control. SOP metrics are characterized by their response to partial predictability, adversarial uncertainty, lookahead, trajectory-level evaluation, and compositional trade-offs between robustness and adaptivity. They arise naturally in algorithms and architectures that interpolate between conventional online and offline settings, in fields such as matching theory, scheduling, learning-augmented online algorithms, LLM agentic benchmarks, recommender systems, neuromorphic hardware, and reinforcement learning.

1. Conceptual Foundation and Definitions

SOP metrics quantify performance where the future contains a mixture of predictable and adversarial elements, or where an agent (or algorithm) is allowed to "rely on its own generated history" over multiple steps rather than resetting to ground truth after each individual decision. In semi-online models, input is partitioned into a predictable subset (for which the algorithm has some prior knowledge, e.g., forecasts, lookahead, or extra parameters) and an adversarial subset revealed online. SOP metrics systematically interpolate between purely offline and purely online settings and are formally defined through competitive ratios, advantage-based rewards, or success rates that capture this interpolation.

For instance, in semi-online bipartite matching (Kumar et al., 2018), let $V = V_P \cup V_A$, where $V_P$ is predictable and $V_A$ adversarial. The competitive ratio is a specific function of the adversarial fraction $\delta = |V_A|/n$, smoothly bridging offline optimality and online lower bounds:

• Integral case: $1 - \delta + \delta^2(1 - 1/e)$
• Fractional case: $1 - \delta e^{-\delta}$

In semi-online reinforcement learning for GUI automation (Lu et al., 15 Sep 2025), SOP is defined as a multi-turn evaluation metric that uses model-generated historical context, terminating only on action mismatch and reflecting progress and task success rate across episodes. This simulates true online interaction dynamics.

2. Mathematical Formulations and Properties

SOP metrics are generally encoded via competitive ratios, success rates, or regression-based mappings, depending on the application domain. Core formulas include:

• Competitive ratio for semi-online bipartite matching:

$$\text{CR}(\delta) = 1 - \delta + \delta^2(1 - 1/e) \quad \text{(integral)}$$

$$\text{CR}(\delta) = 1 - \delta e^{-\delta} \quad \text{(fractional)}$$

• Weighted step and episode-level evaluation (RL for GUI agents) (Lu et al., 15 Sep 2025):

$$\text{Progress (PG)} = \frac{1}{N} \sum_{i=1}^{N} \frac{s_i}{t_i}$$

$$\text{Task Success Rate (TSR)} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}[s_i = t_i]$$

$$\text{SOP Score} = \frac{\text{PG} + \text{TSR}}{2}$$

• Success rate (LLM agents on SOP-guided graphs) (Ye et al., 16 Jan 2025):

$$\text{Success Rate} = \frac{\text{number of successful runs}}{\text{total runs}}$$

Additional metrics (e.g., Pass@1 in code generation, path and leaf accuracy in decision graphs) also reflect SOP in agentic settings.

• Pareto front-based mapping of offline to online metrics in recommender systems (Wilm et al., 13 Jul 2025): Offline metrics along the Pareto front (e.g., Recall@20, OD@20, product Recall@20·OD@20) are used to regress against online KPIs (CTR, CVR, units sold), defining a semi-online correspondence.

These formulas share a common feature: they interpolate or compose between multiple modes of evaluation, often parameterized by the quality or amount of prior information.

3. Evaluation Procedures and Implementation Strategies

SOP metrics are implemented via evaluation protocols that move beyond single-step, ground-truth conditioning. Notable examples include:

• Multi-turn rollouts with patching: In semi-online RL, the agent executes a policy over full trajectories, maintaining its own output, and is evaluated solely on its own generated context unless a critical mismatch occurs, at which point a patch module may inject expert actions if within a threshold (Lu et al., 15 Sep 2025).
• Competitive analysis with adversarial instances and lookahead: Scheduling and matching algorithms are benchmarked by constructing sequences where only partial future information is available (e.g., $k$-lookahead in scheduling (Dwibedy et al., 2023)), and then determining instance-optimal bounds for the competitive ratio.
• Regression of offline proxies against online KPIs: In recommender systems, SOP metric estimation involves splitting traffic among test groups with varied preference vectors, computing offline metrics, and then regressing these metrics against observed online outcomes to establish predictive validity (Wilm et al., 13 Jul 2025).
• Structured decision graph traversal for LLM agents: SOP-agent frameworks evaluate success based on how accurately the agent follows SOP decision graphs using DFS traversal and branch selection grounded in dynamic observation (Ye et al., 16 Jan 2025, Nandi et al., 9 Jun 2025).

Implementation prioritizes the simulation of online dynamics, trajectory-level evaluation, and the systematic incorporation of lookahead, prediction error, or extra operational parameters.

4. Robustness and Interpolation Properties

A central principle of SOP metrics is robust interpolation. Algorithms evaluated by SOP metrics exhibit three foundational behaviors (Kumar et al., 2018, Antoniadis et al., 2020):

1. Consistency: Performance is near-optimal when predictions or lookahead are accurate; SOP metric approaches the offline optimum.
2. Smoothness: Performance degrades continuously as the fraction of unpredictable (adversarial) input increases, or prediction error grows.
3. Robustness: SOP metrics guarantee that performance does not fall below the classical worst-case online bound as predictability diminishes.

For instance, in semi-online matching, the metric satisfies $\text{CR}(0) = 1$ (offline optimum) and $\text{CR}(1) = 1 - 1/e$ (online bound). In SOP evaluation of GUI RL agents, trajectory-level SOP score has a documented high correlation ($R^2=0.934$) with true online evaluation, demonstrating robustness across evaluation regimes (Lu et al., 15 Sep 2025).

5. Practical Applications and Empirical Outcomes

SOP metrics are deployed across a range of domains:

• Matching and scheduling theory: SOP metrics quantify the benefit of partial predictability in combinatorial problems, leading to improved resource allocation and performance guarantees under realistic assumptions.
• Learning-augmented online algorithms: The performance guarantee as a function of prediction error (e.g., $O(\min\{1+\log(1+\eta/\mathrm{Off}), \log k\})$ in caching (Antoniadis et al., 2020)) is a canonical SOP metric, directly informing hybrid algorithm design.
• GUI automation and RL agents: The SOP metric provides an efficient, implementation-friendly proxy for real-world multi-turn performance, with demonstrated state-of-the-art outcomes over competitive baselines (Lu et al., 15 Sep 2025).
• LLM agentic frameworks: SOP metrics (success rate, path and leaf accuracy, Pass@1, TSR, ECR) form the backbone of industrial benchmarking in complex workflows structured by SOPs. Domain-specific benchmarks highlight gaps in agent performance and drive architectural innovation (Nandi et al., 9 Jun 2025, Ye et al., 16 Jan 2025).
• Recommender systems: SOP metrics are central to practitioner strategies that link offline evaluation with online KPIs, enabling cost-effective optimization through Pareto approximations and preference-vector sampling (Wilm et al., 13 Jul 2025).
• Neuromorphic hardware: In SNN processors, SOP (energy per synaptic operation) provides a direct measure of efficiency for event-based learning mechanisms (Frenkel et al., 2018).

6. Extensions, Limitations, and Research Directions

The SOP metric framework is adaptable to new domains, especially where the integration of prediction, lookahead, or partial trajectory is valuable. Research directions include:

• Refinement of SOP metrics in RL domains beyond GUI automation, such as dialogue and mixed-initiative interaction (Lu et al., 15 Sep 2025).
• Extension to settings with partial and noisy information, as in scheduling with approximate sums or partial job parameters (Dwibedy et al., 2020).
• Developing unified SOP metric complexity classes based on the richness of extra input (EPI) (Dwibedy et al., 2020).
• Community-led expansion of SOP-benchmarks for diverse industrial domains, supported by open synthetic data generation pipelines (Nandi et al., 9 Jun 2025).

A plausible implication is that SOP metrics may evolve into standardized evaluation benchmarks for algorithmic systems operating under uncertainty and partial foresight.

7. Formal Summary Table of Representative SOP Metrics

Application Domain	SOP Metric Formula/Type	Key Interpolation/Insight
Bipartite Matching (Kumar et al., 2018)	$1-\delta+\delta^2(1-1/e)$; $1-\delta e^{-\delta}$	Interpolates offline and online extremes
Scheduling (Dwibedy et al., 2023, Dwibedy et al., 2020)	Competitive ratio $CR$ as function of EPI or $k$-lookahead	Lower worst-case load imbalance with minimal lookahead
Reinforcement Learning (Lu et al., 15 Sep 2025)	PG, TSR, SOP Score $(PG+TSR)/2$	Correlates with true online performance
LLM Agents (Nandi et al., 9 Jun 2025, Ye et al., 16 Jan 2025)	Success rate, Pass@1, ECR, TSR, path/leaf accuracy	Structured evaluation of multi-step SOP adherence
Recommender Systems (Wilm et al., 13 Jul 2025)	Regression of offline metrics to online KPIs	Preference-based Pareto front optimization
Neuromorphic Hardware (Frenkel et al., 2018)	$E_{tot,SOP} = P / r_{SOP}$ (energy per synaptic operation)	Application-relevant global energy measure

SOP metrics provide systematic techniques for evaluating algorithmic systems operating at the boundary of online and offline control, establishing robust, theoretically grounded and empirically validated measures of performance. They are critical for interpreting empirical results, designing adaptive algorithms, and advancing practical automation and decision making under uncertainty.