OBELS: Behavioral Leakage Metric

• OBELS is a behavioral privacy metric that structures prompts into triplets (intent, source type, entity) to assess functional leakage beyond surface similarity.
• It decomposes leakage into four dimensions—functional equivalence, domain-type equivalence, semantic equivalence, and entity granularity tolerance—to capture behavioral alignment.
• Empirical results show OBELS recovers over 73% of functional and domain knowledge leakage, outperforming traditional similarity metrics in agent experiments.

Searching arXiv for the cited OBELS paper and closely related research-agent privacy context. OBELS, explicitly expanded as Ontology-aware Behavioral Leakage Scores, is a behavioral privacy metric introduced in "Network-Level Prompt and Trait Leakage in Local Research Agents" (Jeong et al., 27 Aug 2025). It is presented as a custom metric for judging whether a reconstructed prompt is privacy-leaking in the behavioral sense, not just textually similar. In the paper’s formulation, what matters for web and research agents is not whether two prompts look alike on the surface, but whether they would make the agent do the same thing—visit the same kinds of domains, pursue the same goal, and target the same underlying entity or topic. Accordingly, OBELS is described as “a multi-metric scoring scheme that decomposes leakage into dimensions of preserved functionality, domain targets, and semantic entities,” and as “a set of four complementary scores designed to capture behavioral alignment” (Jeong et al., 27 Aug 2025).

1. Behavioral definition and problem setting

OBELS is introduced for Web and Research Agents (WRAs), defined as LLM-based systems that investigate complex topics on the Internet. The motivating claim is that prompt leakage in such systems should not be evaluated by surface similarity alone. The paper states that “in AI agent contexts, leakage risk stems from functional equivalence: a prompt recovery is risky if it causes the agent to visit similar sites, access comparable services, or pursue equivalent goals—regardless of its phrasing” (Jeong et al., 27 Aug 2025).

This framing is specific to agentic browsing. A reconstruction can therefore be privacy-revealing even if it is not textually close to the original prompt, so long as it causes the same browsing behavior. The paper’s example is explicit: rephrasing “Compare Italy and Spain’s digital nomad visas” as “How do work visas differ across southern Europe?” yields low cosine similarity, yet the agent navigates to similar government portals and retrieves overlapping content. This suggests that, in this setting, leakage is tied to downstream action and information-seeking structure rather than lexical overlap.

The paper’s abstract places OBELS at the center of its empirical claims. It states that the metric is used “to comprehensively assess similarity between original and inferred prompts,” and that the attack “recovers over 73\% of the functional and domain knowledge of user prompts” (Jeong et al., 27 Aug 2025).

2. Rationale and contrast with standard similarity metrics

The paper’s main argument is that standard similarity metrics are the wrong tool for prompt leakage in agent settings. OBELS is intended to evaluate functional leakage, not surface similarity. The contrast is made against two baseline families.

For SBERT cosine similarity, the paper states that SBERT captures broad semantic alignment but “suffers from paraphrastic inflation,” meaning it can assign high similarity to prompts that are loosely related, generic, or verbose, even if they would not induce the same behavior in an agent (Jeong et al., 27 Aug 2025).

For LLM-as-Judge, the paper states that such judges are more flexible, but “are prompt-sensitive and often lack interpretability,” producing a single score without indicating what was preserved or lost. By contrast, OBELS is behavior-aware and ontology-aware: it measures whether the reconstructed prompt preserves the same browsing-relevant meaning, even if the wording differs substantially.

The conceptual distinction is therefore not merely methodological. The harmful event is defined as preservation of the task, the source class, and the underlying target of the information-seeking process. A plausible implication is that OBELS is best understood as an evaluation framework for agent-equivalent reconstructions, rather than as a general-purpose text similarity score.

3. Prompt abstraction, triplets, and scoring procedure

The core abstraction is a prompt-to-triplet transformation: $(intent, source\_type, entity)$

Prompts are abstracted into structured semantic triplets that capture the user’s intent, the source type or domain of information/service sought, and the central entity (Jeong et al., 27 Aug 2025). The template also allows extra triplets of type ("target", source_type, entity) when prompts contain modifiers like price, audience, date, location, or purpose.

OBELS is not presented through a closed-form aggregation formula. The paper does not provide a formula such as

$$OBELS = \frac{E_{func}+E_{dom}+E_{sem}+T_{ent}}{4}.$$

Instead, the metric is operationalized through a GPT-4-based evaluation template plus four dimension-specific scores. The text and tables report those scores separately rather than collapsing them into a single combined number (Jeong et al., 27 Aug 2025).

Each dimension is scored from 0.0 to 1.0 using a standardized evaluation template applied to GPT-4, with a greedy, holistic alignment of triplet sets. The scale is:

• 1.0 = completely equivalent
• 0.8 = very similar
• 0.5 = somewhat related
• 0.2 = weakly related
• 0.0 = unrelated or contradictory

The evaluation prompt instructs the model to compare full triplet sets for the original and reconstructed prompts, use semantic similarity rather than string match to align triplets, be “generous in identifying partial matches,” and return a list of aligned triplet pairs, a JSON object with the four scores, and short rationales. The paper places this prompt template in Appendix \ref{app:obels_templates} (Jeong et al., 27 Aug 2025).

4. The four-component scoring scheme

OBELS is a four-part scoring scheme. The paper names the components explicitly as Functional Equivalence (E_func), Domain-Type Equivalence (E_dom), Semantic Equivalence (E_sem), and Entity Granularity Tolerance (T_ent) (Jeong et al., 27 Aug 2025).

Component	What it asks	Example given
E_func	Same high-level task or intent	finding a product; booking a service
E_dom	Same kind of source or service	travel agencies; health databases
E_sem	Semantically aligned entities	“side effects of Prozac” vs. “adverse reactions to fluoxetine”
T_ent	Specificity differences still behaviorally compatible	“Italy” vs. “Rome and Venice”

Functional Equivalence asks whether the original and reconstructed prompts express the same high-level task or intent. The paper’s examples include finding a product, booking a service, comparing options, and planning a task. It is described as the most important dimension because it captures whether the agent would behave similarly.

Domain-Type Equivalence checks whether the prompts engage with the same kind of source or service. The paper’s examples include travel agencies, health databases, policy resources, and visa-process sites. This matters because two prompts may differ in wording but still drive the agent toward the same class of domains.

Semantic Equivalence measures whether the entities themselves are semantically aligned. The canonical example is “side effects of Prozac” versus “adverse reactions to fluoxetine,” which captures semantic equivalence of core topical content.

Entity Granularity Tolerance assesses whether differences in specificity are still compatible with the same underlying task. The paper’s examples include “Italy” versus “Rome and Venice,” and “Honda” versus “Honda Civic.” This dimension recognizes that a reconstructed prompt can be more specific or less specific than the original while still preserving the same behavioral intent.

The scoring template also provides alignment examples such as ["search", "flight", "Europe"] aligned with ["search", "flight", "international flights to Europe"], and ["target", "price_range", "cheapest"] aligned with ["target", "price_range", "low cost"] (Jeong et al., 27 Aug 2025).

5. Role in prompt-leakage experiments and reported results

OBELS is the paper’s principal behavioral privacy score for prompt recovery. The paper states that “attack success is measured not by textual similarity alone, but by semantic and downstream behavior similarity, whether reconstructed prompts preserve the functional intent behind the original.” In the experimental tables, OBELS is reported as four separate dimensions for each attack setting, and is used to evaluate different inference methods, shot counts, agent backbones, LLM inference models, defenses, and trace visibility levels (Jeong et al., 27 Aug 2025).

The flagship 5-shot ICL setup with GPT-4o reports:

• E_func = 0.770
• E_dom = 0.735
• E_sem = 0.520
• T_ent = 0.640

These are the values behind the abstract’s statement that the attack recovers over 73\% of the functional and domain knowledge of user prompts.

The paper reports that ICL generally beats fine-tuning on OBELS dimensions. For GPT-4o, the values are 0.770 / 0.735 / 0.520 / 0.640 for ICL, versus 0.675 / 0.690 / 0.485 / 0.615 for fine-tuning. Across shot counts, 5-shot ICL is presented as the strongest overall balance, with the following sequence: 0-shot: 0.665 / 0.630 / 0.465 / 0.615; 5-shot: 0.770 / 0.735 / 0.520 / 0.640; 8-shot: 0.760 / 0.730 / 0.540 / 0.610; 12-shot: 0.750 / 0.710 / 0.530 / 0.640; 15-shot: 0.770 / 0.685 / 0.525 / 0.645.

Across agents on SESSION14, the paper reports: AutoGen: 0.565 / 0.625 / 0.400 / 0.495; Browser-Use: 0.540 / 0.590 / 0.390 / 0.485; GPT-Researcher: 0.770 / 0.735 / 0.520 / 0.640; Deep Research: 0.680 / 0.720 / 0.480 / 0.640. The authors interpret GPT-Researcher as leaking the most.

Across trace visibility, the reported values are domain-only traces: 0.770 / 0.735 / 0.520 / 0.640; full URLs: 0.810 / 0.805 / 0.620 / 0.695; partial URLs at 80\% visibility: 0.795 / 0.795 / 0.635 / 0.680; partial URLs at 60\% visibility: 0.820 / 0.755 / 0.585 / 0.690. The paper’s interpretation is that more visibility increases leakage, but even reduced visibility retains substantial OBELS scores.

Under defenses, the paper reports 5 decoys: 0.565 / 0.590 / 0.395 / 0.500, and 5 decoys + shuffle: 0.650 / 0.695 / 0.465 / 0.570. The stated conclusion is that defenses reduce OBELS, but not to zero.

6. Interpretation, thresholds, and limitations

The paper’s interpretive frame is that OBELS should be read as a decomposition of privacy leakage into distinct behavioral dimensions, rather than as a single summary number. The authors emphasize decomposition because a single score can hide where alignment fails. That design choice increases interpretability, but it also means OBELS requires reading multiple numbers rather than one simple scalar (Jeong et al., 27 Aug 2025).

The paper does not define dedicated OBELS thresholds. It notes threshold-like language elsewhere in the paper for trait-inference plots: scores above 0.7 are described as high-risk leakage, and figures mark 0.5 and 0.7 as moderate/high-risk thresholds. The text is explicit that this thresholding is for trait inference plots, not OBELS itself. A plausible implication is that OBELS values are intended primarily for comparative analysis across settings rather than for a fixed universal decision rule.

The main caveat is that OBELS is still an LLM-judged metric, and therefore inherits some subjectivity. The paper’s implied limitations are specific: it relies on GPT-4-based evaluation, so scores depend on the judge model and prompt template; it is structured and interpretive, not a deterministic mathematical metric; it evaluates behavior via a semantic abstraction layer, so it may miss some finer-grained or latent behavioral differences; it is designed for prompt recovery in WRA contexts, not presented as a universal metric for all similarity evaluation tasks; and it measures whether prompts would lead to similar browsing behavior, but not necessarily all downstream consequences. The authors also note that prompts are abstracted into triplets and scored greedily, which makes the metric dependent on the quality of triplet extraction and alignment (Jeong et al., 27 Aug 2025).

Within the paper’s overall argument, OBELS functions as the central evidentiary link between network-level metadata and privacy harm. Its significance lies in formalizing the claim that, for local research agents, the salient leakage event is preservation of behavioral alignment—the same task, the same source class, and the same core entity—even when textual reconstruction remains imperfect.