IDEA2: Expert-in-the-Loop CQ Elicitation

• IDEA2 is a semi-automated workflow that integrates LLM-based extraction with expert validation to elicit competency questions.
• It employs an iterative reformulation process and consensus-driven reviews to refine, prioritize, and validate questions effectively.
• The method ensures full provenance tracking using standards like PROV-O and OWLUnit, enhancing reproducibility and auditability.

IDEA2 is a semi-automated, expert-in-the-loop workflow for competency question elicitation in collaborative ontology engineering. It addresses a phase described as a critical but resource-intensive bottleneck: the translation of requirements held by domain experts into formalizable competency questions (CQs) by ontology engineers. The method integrates LLMs into an iterative process that begins with CQ extraction from requirement documents, proceeds through collaborative expert review on an accessible platform, and reformulates rejected CQs until consensus is achieved, while preserving full provenance for transparency and reproducibility (Watkiss-Leek et al., 1 Apr 2026).

1. Problem setting and conceptual scope

Competency question elicitation is presented as foundational in ontology engineering, yet often constrained by a communication gap between domain experts, who possess the relevant domain knowledge, and ontology engineers, who formalise that knowledge. IDEA2 is designed for new ontology engineering projects and targets this gap by combining LLM-based generation with expert validation rather than replacing either role (Watkiss-Leek et al., 1 Apr 2026).

The workflow operates on heterogeneous requirement sources. Its extraction phase accepts unstructured user stories, personas, and use-case narratives, as well as structured schema definitions such as XML. Two properties are enforced in the initial CQ generation step: atomicity, through decomposition of complex questions into atomic units, and abstraction, through replacement of named entities with conceptual placeholders. Duplicate removal is then performed through sentence-embedding similarity thresholding.

A central feature of the method is that validation is collaborative and explicit. Candidate CQs are surfaced to experts, who do not merely edit text informally; they vote, tag, prioritise, and comment. This makes CQ elicitation a managed lifecycle rather than a one-shot prompt-generation task. A plausible implication is that IDEA2 treats requirements engineering as an iterative consensus process with auditable state transitions, rather than as a static documentation exercise.

2. Four-phase workflow architecture

The architecture is organised into four tightly coupled phases: multi-source LLM-based CQ extraction, collaborative expert validation, iterative LLM reformulation, and provenance tracking with export (Watkiss-Leek et al., 1 Apr 2026).

In the extraction phase, an LLM prompt chain applies a few-shot “decompose & generalise” strategy. The prompt instructs the model to identify potential competency questions, split complex queries into atomic ones, and replace specific names with abstract concepts. This phase is intended to convert heterogeneous requirement materials into a CQ pool that is already partially normalised for ontology engineering use.

In the validation phase, candidate CQs are published via the Notion API as database entries in a Notion dashboard. Experts vote “accept/reject,” assign structured tags such as “ambiguous” or “too specific,” add priorities through “Buy-a-Feature,” and provide free-text comments. Each CQ receives a consensus score

$s = (\#\text{accepts}) - (\#\text{rejects}) .$

This score determines whether a CQ is accepted directly or re-queued for revision.

In the reformulation phase, all CQs with $s \le 0$ are automatically re-queued. The reformulation prompt re-injects the original source text, the prior CQ draft, the vote tally, and anonymised expert comments. The LLM produces a revised CQ, which is republished for another validation cycle. The loop continues until a stopping criterion is met: majority acceptance, iteration cap, semantic-stability threshold, or resource quota.

In the final phase, every CQ is exported with version history, feedback, and generation parameters. The provenance output is JSON-LD conforming to OWLUnit, PROV-O, and the Croissant data model. This export layer is integral rather than ancillary: it is part of the system’s claim to transparency and reproducibility.

3. Provenance model and lifecycle formalisation

IDEA2 models the lifecycle of each competency question as a PROV-O inspired directed acyclic graph. The graph distinguishes entities $E = \{CQ^1, CQ^2, \dots\}$, activities $$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$, and agents $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$ (Watkiss-Leek et al., 1 Apr 2026).

Each versioned CQ is represented as

$$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$

where $\mathit{id}$ is a UUID, $$\mathit{origin}\in\{\text{“LLM-Extracted”},\text{“Human-Authored”}\}$$, $\mathit{sourceDoc}$ is a URI to the input specification, $$\mathit{config}=\langle \text{model}, \text{checkpoint}, \text{seed}, T, p, f\rangle$$ stores generation parameters, $s \le 0$0 is the version number, and $s \le 0$1 records structured feedback for that version.

The provenance graph uses edges such as prov:wasGeneratedBy, prov:wasDerivedFrom, and prov:wasAttributedTo to connect CQ versions to activities and agents. The explicit version lineage means that a reformulated CQ is not treated as an overwrite of an earlier draft but as a derived entity with preserved ancestry. This design supports exact re-runs or audits because the graph captures the model checkpoint, random seed, temperature, top-$s \le 0$2, frequency penalty, and associated human judgments.

The paper’s emphasis on provenance also clarifies the epistemic status of the output. Accepted CQs are not merely LLM completions; they are traceable artefacts whose state reflects interactions among source documents, prompt configuration, expert feedback, and successive reformulations.

4. Iterative reformulation loop and generation configuration

The core algorithm begins with LLM_Extract_CQs(S), deduplicates the extracted set, publishes the current pool to the dashboard, collects expert feedback, accepts all CQs with positive score, and reformulates the remainder until convergence or a stopping condition is reached (Watkiss-Leek et al., 1 Apr 2026).

Operationally, the accepted set is defined by

$s \le 0$3

while the rejected set is passed into a reformulation routine. The pseudocode indicates three convergence controls: absence of rejected CQs, a maximum-iteration bound, and a convergence threshold based on the ratio $s \le 0$4. This makes the loop neither unconstrained nor purely heuristic; it is an iterative process with explicit termination logic.

Both extraction and reformulation use Gemini 2.5 Pro over Google Cloud API with a 1 M-token context window. The generation parameters captured in each CQ’s configuration include temperature $s \le 0$5, top$s \le 0$6, frequency_penalty $s \le 0$7, random seed $s \le 0$8, and model checkpoint $s \le 0$9 “gemini-2.5-pro”. The extraction prompt asks for potential CQs, atomic decomposition, and abstraction of specific names. The reformulation prompt includes the previous CQ draft, expert votes, comments, and a relevant source excerpt, and asks the model to resolve ambiguity and align with domain terminology.

A common misunderstanding would be to treat IDEA2 as a fully autonomous CQ generator. The documented workflow instead assigns distinct roles to the LLM and to domain experts: the model proposes and revises, while experts validate, critique, and determine acceptance.

5. Evaluation scenarios and quantitative outcomes

IDEA2 was validated in two real-world scenarios: a full end-to-end scientific-data study based on AnIML and a cultural-heritage reformulation stress test based on pre-rejected questions from the AskCQ dataset (Watkiss-Leek et al., 1 Apr 2026).

In the AnIML scenario, $E = \{CQ^1, CQ^2, \dots\}$0 domain experts from Unilever PLC evaluated CQs derived from the AnIML XML core schema, approximately 2,500 lines, plus technique definitions. The study exercised the entire workflow from extraction through three iterations of review and reformulation to a final CQ set, and validation included SPARQL queries against an OWL ontology. In the cultural-heritage scenario, $E = \{CQ^1, CQ^2, \dots\}$1 experts evaluated 23 pre-rejected CQs, 11 human-authored and 12 LLM-generated, and only the reformulation loop was exercised for two iterations.

The principal metrics were acceptance rate per iteration,

$E = \{CQ^1, CQ^2, \dots\}$2

cumulative final acceptance,

$E = \{CQ^1, CQ^2, \dots\}$3

average time per iteration and per CQ, and inter-annotator agreement metrics including observed agreement $E = \{CQ^1, CQ^2, \dots\}$4, Krippendorff’s $E = \{CQ^1, CQ^2, \dots\}$5, Fleiss’ $E = \{CQ^1, CQ^2, \dots\}$6, Gwet’s AC1, PABAK, and pairwise Cohen’s $E = \{CQ^1, CQ^2, \dots\}$7 range.

Scenario	Setup	Reported outcome
AnIML	4 experts; 103 initial CQs; 3 iterations	$E = \{CQ^1, CQ^2, \dots\}$8 final acceptance $E = \{CQ^1, CQ^2, \dots\}$9; $$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$0 min/iteration; $$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$1 min/CQ
Cultural Heritage	3 experts; 23 initial CQs; 2 iterations	$$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$2 final acceptance $$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$3; $$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$4 min/iteration; $$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$5 min/CQ

Iteration-wise results show different dynamics in the two domains. In AnIML, iteration 1 evaluated 103 CQs with $$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$6 acceptance at $$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$7 min/CQ, iteration 2 evaluated 5 CQs with $$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$8 acceptance at $$A = \{\text{Extract}, \text{Reformulate}_i, \text{Validate}_i\}$$9 min/CQ, and iteration 3 evaluated 2 CQs with $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$0 acceptance at $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$1 min/CQ. In cultural heritage, iteration 1 evaluated 23 CQs with $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$2 acceptance at $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$3 min/CQ, and iteration 2 evaluated 7 CQs with $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$4 acceptance at $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$5 min/CQ. This suggests that later iterations concentrate harder cases, increasing per-item deliberation cost even as the candidate pool shrinks.

Inter-annotator agreement reveals an important statistical nuance. Observed agreement was $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$6 in AnIML and $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$7 in cultural heritage, but chance-corrected measures differed substantially: Krippendorff’s $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$8 was $Ag = \{\text{LLM}, \text{HumanExpert}_k\}$9 for AnIML and $$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$0 for cultural heritage; Fleiss’ $$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$1 was $$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$2 and $$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$3, respectively. By contrast, Gwet’s AC1 was $$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$4 and $$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$5, and PABAK was $$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$6 and $$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$7. Pairwise Cohen’s $$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$8 ranged from $$CQ^{(v)} = \bigl\langle\,\mathit{id},\,\mathit{origin},\,\mathit{sourceDoc},\,\mathit{config},\,v,\,\mathit{feedback}_v\,\bigr\rangle ,$$9 in AnIML and $\mathit{id}$0 in cultural heritage. The paper explicitly interprets this as a prevalence effect under high-acceptance imbalance.

6. Usability, methodological significance, and limitations

The paper reports four 5-point Likert-scale usability questions covering clarity, collaboration support, expressiveness, and intuitiveness, and Figure 1 shows that in both domains more than $\mathit{id}$1 of responses were “Agree” or “Strongly Agree” on intuitiveness, collaboration support, and expressiveness (Watkiss-Leek et al., 1 Apr 2026).

Several methodological conclusions are drawn. First, iterative reformulation is described as critical: providing structured expert comments yields targeted, semantically accurate CQ revisions, exemplified by refining “result blueprints” to “names of the result definitions specified by a technique.” Second, the workflow induces a role shift in requirements engineering. Knowledge engineers set up prompts and monitor the system, while domain experts become active co-creators and validators, reducing manual CQ authoring effort. Third, embedding the process in a familiar, cloud-native platform, specifically Notion, is reported to reduce technical friction and support asynchronous, distributed teams.

The reproducibility claims are unusually explicit. Every CQ’s lineage includes seed, temperature, model checkpoint, and expert feedback, and is exportable as standard JSON-LD using PROV-O, OWLUnit, and Croissant. Code and experiments are released at the project repository indicated in the paper. In this respect, IDEA2 treats LLM-generated artefacts as versioned research objects rather than transient interface outputs.

The empirical evidence also bounds the claims that can be made. The workflow was validated in two real-world scenarios, one full end-to-end and one reformulation-only stress test. The reported outcomes support acceleration of requirements engineering, improved acceptance and relevance of resulting CQs, and high usability and effectiveness among domain experts. At the same time, the agreement analysis shows that raw agreement near $\mathit{id}$2 does not by itself establish uniform annotator consistency; the paper therefore highlights the need to interpret $\mathit{id}$3, $\mathit{id}$4, AC1, and PABAK together rather than relying on a single statistic. This is a methodological caution rather than a defect: IDEA2 formalises expert disagreement as part of the elicitation process and records it as provenance-bearing evidence.