Expert-in-the-Loop Probing Pipeline

• Expert-in-the-loop probing pipelines are workflows that integrate human expert judgments with computational probing to systematically refine hypotheses.
• They employ conditional gating, selective review, and iterative narrowing techniques to enhance variable selection and debugging across diverse domains.
• These pipelines emphasize traceable provenance and controlled feedback loops, balancing automated search with expert decision-making.

Searching arXiv for papers relevant to expert-in-the-loop probing pipelines. An expert-in-the-loop probing pipeline is a workflow in which a computational system probes a search, hypothesis, decision, or explanation space, while expert knowledge is injected as a conditioning signal, an adjudication mechanism, or a refinement driver. In the cited literature, this pattern appears in post-silicon variable selection, information-extraction debugging, competency-question elicitation, ancient intertextual analysis, and enterprise decision-rule refinement. Despite domain differences, the recurrent structure is stable: candidate generation or exposure, selective expert review, routed feedback, and iterative narrowing of the space of acceptable outputs rather than fully autonomous inference (Liao et al., 2022, Sarma et al., 2010, Watkiss-Leek et al., 1 Apr 2026, Umphrey et al., 2024, Zha et al., 9 Jun 2026).

1. Core architectural pattern

The central architectural property is not merely human oversight, but conditional probing. In "Experts in the Loop: Conditional Variable Selection for Accelerating Post-Silicon Analysis Based on Deep Learning" (Liao et al., 2022), the input is explicitly partitioned as

$X=[X_p, X_c],$

where $X_p$ contains preselected variables supplied by experts and $X_c$ contains candidate variables whose relevance is learned only after conditioning on that expert-defined context. The learned mask $\bm m$ is therefore not a generic feature-importance vector; it is a conditional gating policy over $X_c$.

In "PROBER: Ad-Hoc Debugging of Extraction and Integration Pipelines" (Sarma et al., 2010), the probing target is not a feature set but the causal structure of an IE pipeline. The system performs post-execution analysis over a DAG of heterogeneous operators and returns compact provenance objects sufficient for debugging suspicious records rather than complete provenance for the entire run. The design is explicitly “ad-hoc” and “pay-as-you-go,” meaning that the depth of provenance construction is driven by the user’s need.

In "IDEA2: Expert-in-the-loop competency question elicitation for collaborative ontology engineering" (Watkiss-Leek et al., 1 Apr 2026), the pipeline separates generation from validation. An LLM extracts candidate competency questions from requirement material, but domain experts remain the final arbiters of relevance, clarity, abstraction level, and correctness. Rejected questions are not discarded; they are routed back into a reformulation loop.

In "Investigating Expert-in-the-Loop LLM Discourse Patterns for Ancient Intertextual Analysis" (Umphrey et al., 2024), the LLM is used as a candidate-retrieval and analysis aid rather than an autonomous classifier. The expert defines the scenario, evaluates retrieved candidates with Hays’ criteria, and may iteratively alter prompt scope or passage segmentation.

In "Trace2Policy: From Expert Behavior Traces to Self-Evolving Decision Agents" (Zha et al., 9 Jun 2026), the optimization target is a human-readable rule document rather than a latent model state alone. The system executes current rules, diagnoses errors, proposes targeted patches, and advances only regression-safe revisions.

2. Probe space construction and selective focus

A defining feature of these pipelines is that they do not probe the full hypothesis space uniformly. They impose explicit structures that make expert review tractable.

In CVS, the expert-defined split $X=[X_p,X_c]$ reduces the problem from unconstrained variable selection to conditional ranking. Candidate variables are masked by a learnable vector

$\bm m=[m_1,\dots,m_{D_c}]^T,\qquad m_i\in(0,1),$

and the top-$k$ variables after training are the candidate variables with the largest entries in $\bm m$ (Liao et al., 2022). The paper’s synthetic and industrial experiments show why this matters: a variable may be important or unimportant depending on which variables are already known, and CVS can suppress candidates that are redundant with the preselected context.

PROBER constructs focus through the notion of an MISet, a minimal subset of input records sufficient to produce a given output:

$$I_s\subseteq I \text{ is an MISet for } r \iff r\in O(I_s)\ \text{and}\ \forall I'\subset I_s,\ r\not\in O(I').$$

This changes the debugging unit from “everything that might matter” to the smallest record subsets that actually witness production of an output record (Sarma et al., 2010). The system also provides all-, any-, impact-, union-, and intersection-provenance, allowing the user to begin with coarse summaries and request richer provenance only when necessary.

IDEA2 narrows the probe space at the text level. The extraction prompt uses few-shot learning and a multi-step refinement chain to enforce atomicity and abstraction: complex questions are split into atomic CQs, and instance-level references are generalized to conceptual terms. Deduplication by sentence embedding similarity further reduces redundancy before expert review (Watkiss-Leek et al., 1 Apr 2026).

In the intertextual-analysis workflow, the probe space is defined by explicit corpus/query passage pairs in Koine Greek and by a fixed prompt asking for quotations, allusions, general dependence, and shared rare words. The expert then scores candidate links with fuzzy weights—low, mid, or high—rather than treating all returned links as equally plausible (Umphrey et al., 2024).

3. Feedback routing, gating, and iterative refinement

The loop becomes substantive only when expert judgments change subsequent computation. The cited systems implement this through routing rules and gating policies.

IDEA2 uses a binary expert judgment—accept or reject—aggregated into a numerical consensus score. If a CQ fails to achieve a positive net score $X_p$0, it is automatically sent back for reformulation. The reformulation prompt is highly contextualized: it includes the original requirement specification, the original CQ, the validation log, scores and votes, anonymized qualitative comments, and version history. The loop repeats until a stopping criterion such as convergence threshold, iteration cap, modification stability, or resource quota is met (Watkiss-Leek et al., 1 Apr 2026).

Trace2Policy formalizes its refinement engine as

$X_p$1

with the sequence Execute $X_p$2 Compare $X_p$3 Diagnose $X_p$4 Refine. Diagnose assigns each wrong validation case to one of three root-cause types: MISSING, WRONG, or CONFLICT. Patches are committed only if they pass a regression gate; if action accuracy drops by more than 2%, the patch is rolled back and the pre-round snapshot is restored (Zha et al., 9 Jun 2026). The mechanism is therefore not free-form iterative editing, but controlled policy revision under explicit rollback constraints.

PROBER’s pay-as-you-go logic is a related but non-optimization-centered form of routing. A user can ask first for only a small number of explanations, inspect provenance for intermediate or final records, and progressively refine from summaries such as intersection-provenance to more expensive views such as all-provenance. The system can start returning MISets as soon as they are found rather than waiting for full enumeration (Sarma et al., 2010).

The intertextual-analysis pipeline uses a lighter, human-guided prompting loop. After reviewing initial outputs, the expert may narrow the corpus, split long passages, or rerun the prompt with altered context. The paper is explicit that this is not a formal training loop; it is an expert-guided refinement loop used to stabilize candidate retrieval and interpretive judgment (Umphrey et al., 2024).

System	Probe target	Routing mechanism
CVS	Candidate variables conditioned on $X_p$5	Rank by learned mask $X_p$6
PROBER	Suspicious output and intermediate records	Progressive provenance queries
IDEA2	Candidate competency questions	Positive vs. non-positive consensus score
Trace2Policy	Validation errors in rule execution	MISSING / WRONG / CONFLICT + regression gate
Intertextual analysis	Candidate intertextual links	Expert low / mid / high weighting and prompt revision

4. Provenance, auditability, and interpretability

A notable convergence across these systems is that expert-in-the-loop probing is rarely treated as a black-box heuristic. It is coupled to explicit provenance or human-readable artifacts.

PROBER is the most formal instance. It defines a provenance model that supports exact, IO, constraint-based, and black-box operators; proves composition results for monotonic operators; and distinguishes tractable from intractable provenance tasks. Computing all MISets or union-like provenance can be #P-complete in general, but bounded MISet size yields tractable regimes on the order of roughly $X_p$7 (Sarma et al., 2010). Interpretability is therefore tied to formal operator properties such as monotonicity, one-to-one, one-to-many, and many-to-one.

IDEA2 tracks the full lifecycle of each CQ through a provenance record containing a unique identifier, origin, link to the source specification, generation configuration, version history, and review-cycle outcomes. The generation configuration records settings such as model checkpoint, seed, temperature, TopP, and frequency penalty. Export uses JSON-LD, combining OWLUnit for CQ definition, PROV-O for lineage, and Croissant for dataset structure (Watkiss-Leek et al., 1 Apr 2026). The result is a versioned lineage graph rather than a flat list of accepted questions.

Trace2Policy likewise treats the policy artifact as auditable infrastructure. The “Skills” document is human-readable, version-controlled, and executable either as an LLM prompt or as compiled deterministic Python. Because refinement operates through targeted edits to this rule document, the system preserves an explicit relation between observed failure clusters and concrete policy patches (Zha et al., 9 Jun 2026).

CVS does not use a provenance ontology, but it does provide a directly interpretable artifact: the learned mask $X_p$8, whose entries can be sorted to expose the top-$X_p$9 candidate variables conditioned on expert-selected variables (Liao et al., 2022). In this setting, interpretability is rank-based rather than lineage-based.

5. Domain-specific realizations

The general architecture manifests differently across domains, but each instance makes the same trade: computational breadth for expert-directed selectivity.

In post-silicon validation, CVS was evaluated on synthetic data with 2000 training samples and 15 input variables, and on an industrial dataset with 9 DUTs, 13 input variables, 1 FoM target, and 900,000 total test cases with 100,000 per DUT (Liao et al., 2022). The paper reports that the learned mask converged in about 4k epochs and roughly 5 minutes on a consumer GPU, and that the model can avoid selecting variables redundant to preselected ones. It also highlights a combinatorial advantage over exhaustive search: selecting the best 5 variables out of 10 candidates would require $X_c$0 combinations for one DUT, whereas CVS produces importance scores in a single training run.

In IE debugging, PROBER was evaluated on large-scale web extraction, including 500 million web pages crawled by Yahoo and a business pipeline built from 5,443,183 web pages from 147 sites (Sarma et al., 2010). The paper reports that MISet-based provenance corresponds to the smallest possible document set and describes a debugging example where an erroneous business record was traced back to a bad segmentation operator.

In ontology engineering, IDEA2 was validated in two real-world scenarios. In the AnIML scenario, the workflow used 4 domain experts, started from 103 initial candidate CQs, ran for 3 iterations, and achieved a 92.7% final acceptance rate with average time per CQ of 0.85 min and average time per iteration of 10.78 min. In the cultural-heritage scenario, it used 3 cultural heritage experts, began from 23 rejected CQs, ran for 2 iterations, and achieved a 73.3% final acceptance rate with average time per CQ of 2.97 min (Watkiss-Leek et al., 1 Apr 2026).

In ancient intertextual analysis, the evaluation comprised 6 scenario sets spanning true positive, false positive, probable, speculative, hypothesis, and out-of-sample conditions. The paper’s most explicit procedural conclusion is that query length matters significantly, whereas corpus length is less problematic when it is roughly in the range of 1–3 chapters (Umphrey et al., 2024). The workflow succeeds at retrieving known quotations and detecting lexical, morphological, structural, and thematic correspondences, but it can also overcall direct dependence when a shared pretext is the better explanation.

In enterprise decision automation, Trace2Policy begins by recording expert work traces, transforming them into structured decision records, and distilling them into an initial rule set. In the logistics case study, this yielded 555 trajectories, 476/555 traces as usable structured records, and an initial policy with 62 rules, 4 decision paths, 41 judgment situations, and 17 outcome codes (Zha et al., 9 Jun 2026). Human EISR ran for 8 rounds. The paper reports that one-shot distillation plateaus near ~70% on the deployed pool, while eight EISR rounds lift the same rules to 79.6% when compiled into deterministic Python. The system was deployed for 22 days on 3,349 audit cases, and the compiled pipeline outperformed the pure-LLM baseline of 72.7%.

6. Limitations, boundary cases, and conceptual distinctions

The literature is consistent in rejecting a common misconception: expert-in-the-loop probing is not presented as expert replacement. CVS explicitly states that post-silicon analysis is inherently a human-in-the-loop process and that the method should help experts focus on the most relevant variables rather than replace them (Liao et al., 2022). IDEA2 similarly separates generation from validation, leaving final authority with domain experts (Watkiss-Leek et al., 1 Apr 2026). The intertextual-analysis paper makes the same point by treating the LLM as a finder and the scholar as validator and contextualizer (Umphrey et al., 2024).

The approach also inherits domain-specific limits. PROBER mainly focuses on monotonic operators; non-monotonic debugging is only sketched via MASet, and complete provenance can be computationally infeasible because all-MISet and union-like inference may be #P-complete (Sarma et al., 2010). IDEA2 depends on consensus formation and on expert willingness to provide structured and qualitative feedback; the paper therefore defines several stopping criteria rather than assuming guaranteed convergence (Watkiss-Leek et al., 1 Apr 2026). The intertextual pipeline is vulnerable to long-query instability, false intertextual dependences, and failure to distinguish direct dependence from shared background tradition (Umphrey et al., 2024). Trace2Policy shows that even after refinement, execution form matters: the same EISR-refined content performs 9.8 percentage points higher as compiled Python than as an Opus prompt in production, and re-enabling LLM fallback monotonically degrades accuracy on the reported skewed-base-rate workloads (Zha et al., 9 Jun 2026).

A further boundary case appears in "LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls" (Zhang et al., 12 Nov 2025). That paper presents a closed-loop probing pipeline in which Greedy Capability Probing, Judgement-Guided Label Verification, and Error-Driven Data Expansion are driven by models rather than human experts. The paper explicitly characterizes this as judge-model-in-the-loop / self-refining automated loop, not human expert-in-the-loop. This suggests that recent work is broadening the architecture from direct human intervention to stronger automated verification, while preserving the same structural logic of probe, diagnose, route, and refine.

Taken together, these systems define an expert-in-the-loop probing pipeline as a rigorously selective methodology: the system probes a complex space, but experts determine what constitutes context, suspicion, validity, or acceptable revision. The resulting pipeline is neither purely manual nor purely autonomous. It is a controlled interaction between computational search and expert judgment, with explicit routing rules, provenance or human-readable artifacts, and domain-specific mechanisms for keeping the search space both tractable and epistemically grounded.