Human-Assisted LLM Mediation

• Human-assisted LLM mediation is a framework that uses large language models to structure unstructured data while incorporating human corrections in real time.
• It employs a closed-loop architecture combining real-time input processing, LLM summarization, and targeted human interventions to prevent recurring errors.
• Empirical evaluations using metrics like DER and ImDER show significant error reduction and improved system performance with minimal user involvement.

Human-assisted LLM mediation refers to workflows and systems in which LLMs serve as intermediaries—actively supporting communication, conflict resolution, decision-making, or information management—while the ultimate oversight, critical corrections, or key judgments remain with human users. Unlike fully automated systems, human-assisted approaches treat LLMs as adaptive, interactive agents that bridge cognitive, procedural, or relational gaps between stakeholders without supplanting human agency. Recent research formalizes such mediation as a tight loop in which LLMs (a) distill or structure raw input, (b) accept and translate human interventions into actionable updates, and (c) use these updates to prevent future errors or drift, realizing a feedback-stabilized, real-time, and user-centric AI workflow (He et al., 22 Sep 2025).

1. Core Architecture and Mediation Loop

A prototypical human-assisted LLM mediation system comprises multiple tightly integrated modules operating in a continual feedback loop:

• Input Layer: Designed for real-time, streaming ingestion (e.g., ASR-transcribed speech) (He et al., 22 Sep 2025), batch dispute documents (Pawar et al., 2 Sep 2025), or free-text communications.
• LLM Mediation Layer: The LLM is tasked with extracting, summarizing, or interpreting unstructured input in a user-friendly form (e.g., concise per-speaker meeting summaries; structured dispute schemas).
• Feedback Layer: Human users provide brief, targeted interventions—typically natural language corrections or clarifications—which are surfaced through lightweight triggers (e.g., wake-words for spoken corrections, in-chat flags for ODR).
• LLM-Driven Update/Correction Layer: The LLM parses human feedback, localizes relevant segments or actions, and issues structured corrections (e.g., JSON blobs) that can be directly incorporated by downstream modules.
• Online Adaptation: Corrected segments or user feedback immediately update the system state—for example, by refining speaker embedding pools for diarization or adjusting constraint sets to prevent error recurrence.

This architecture allows LLMs to act as workflow mediators, transforming open-loop, black-box pipelines into closed-loop, user-stabilized systems that adapt over time (He et al., 22 Sep 2025, Dranias et al., 31 Mar 2026).

2. Correction Incorporation and Online Adaptation

Human-assisted LLM mediation critically depends on efficient, minimally disruptive user input channels and immediate online adaptation:

• Verbal Correction & Parsing: In interactive settings (e.g., live meeting diarization), a wake-word detector (e.g., “Hey Cobi”) captures short user utterances that specify errors, e.g., “Speaker B did not say that; it was Speaker D”. These are parsed by the LLM into a structured JSON response:

  { "original_speaker_id": "B", "original_sentence": "...", "corrected_speaker_id": "D", "corrected_sentence": "..." }

  (He et al., 22 Sep 2025)
• Direct Segment Relabeling and Enrollment: The system re-labels the affected segment(s) and appends the corrected segment's features to the relevant prototype pool, yielding an incremental speaker model update. This process runs asynchronously and does not stall the main data flow.
• Workflow-Specific Update Equations: For diarization,

$$\text{score}(s,i) = \max_{j=1\ldots m_i} \frac{e_s^\top e_i^j}{\|e_s\|\;\|e_i^j\|}$$

segments are dynamically re-evaluated based on the enriched prototype pool following each user correction (He et al., 22 Sep 2025).

• Prompt Engineering for Robust Feedback Integration: Prompts to the LLM are carefully templated for summarization, feedback parsing, and correction; output is strictly structured to minimize misinterpretation.

Online enrollment ensures corrections have persistent, compounding effect, dynamically preventing repeat errors in subsequent model predictions. Sensitivity analyses demonstrate that most of the benefit saturates with one or two reenrollments per user, optimizing the balance between workload and error rate (He et al., 22 Sep 2025).

3. Enhanced Mediation through Split-When-Merged and Summarization

To maximize both human and LLM corrective efficacy, dedicated pre-processing and summarization subroutines are employed:

• Split-When-Merged (SWM) Algorithm: ASR output segments containing potential rapid speaker switches are algorithmically split via word-level sliding window voting, dominance check, and a combinatorial split-point search (Algorithm 1). Only candidate splits in which each new segment is majority-labeled as a distinct speaker, and neither is empty, are accepted:

  1. For sliding windows across the ASR segment, predict speaker on each. 2. Assign speaker to nearest word, bin results. 3. If one speaker is ≥θ dominant, do not split. 4. Else, find i* maximizing separation; accept only distinct-majority splits.

  This is $O(n\cdot|speakers|)$ per segment and avoids both human and LLM confusion induced by merged multi-speaker utterances (He et al., 22 Sep 2025).
• LLM Summarization: Instead of full transcripts, users interact with per-speaker, interval-based summaries (~30–40 words) generated by the LLM, which improves cognitive scan-ability and correction accuracy. Display mode ablations show that summary mode outperforms conversation mode by a factor of >3 in diarization error reduction (He et al., 22 Sep 2025).

These pre-processing submodules increase the overall efficiency and effectiveness of human-in-the-loop corrections by proactively structuring both the data surface humans see and the granularity of candidate correction spans.

4. Evaluation Protocols and Quantitative Benefits

Rigorous, domain-specific metrics anchor claims of mediation benefit:

• Diarization Error Rate (DER):

$$\text{DER} = \frac{T_{\mathrm{miss}} + T_{\mathrm{fa}} + T_{\mathrm{conf}}}{T_{\mathrm{total}}} \times 100\%$$

With T_{\mathrm{miss}}, T_{\mathrm{fa}} fixed by ASR, improvements primarily derive from reductions in T_{\mathrm{conf}} (speaker confusion).

• Relative improvement:

$$\mathrm{ImDER} = \frac{\mathrm{DER}_{\mathrm{baseline}} - \mathrm{DER}_{\mathrm{system}}}{\mathrm{DER}_{\mathrm{baseline}}} \times 100\%$$

• Empirical Results (AMI Test Set) (He et al., 22 Sep 2025):

| System | DER (%) ↓ | SErr (%) ↓ | ImDER (%) ↑ | ImSErr (%) ↑ | |--------------------------|-----------|------------|-------------|--------------| | Baseline | 68.40 | 15.34 | — | — | | Baseline + SWM | 67.45 | 14.36 | 1.39 | 6.22 | | SWM + OE + LLM (proposed)| 61.62 | 8.56 | 9.92 | 44.23 |

• Correction Frequency and Online Enrollment Sensitivity: Error rates plateau after 1–2 online enrollments per correction, and the optimal feedback interval is approximately every 2–5 minutes.

These quantitative findings corroborate that the human–LLM closed loop reduces confusional speaker assignments by nearly half over baselines, with improvements concentrated in cognitive-load-efficient (summary) display settings.

5. Prompting, Interface Design, and Human–LLM Synergy

The mediation efficacy is strongly determined by prompt engineering, interface design, and the ease of interaction provided to users:

• Prompt Templating (LLM): All instructions to the LLM—summarization, feedback parsing, correction output—are delivered in tightly constrained prompts with JSON output expected for modifications.
• Interface Coupling: The ASR and diarization pipelines run uninterrupted while human feedback and LLM-based correction routines execute asynchronously, ensuring real-time usability.
• User Interaction Minimalism: Verbal corrections are brief, naturalistic utterances triggered by a lightweight phrase detector, minimizing disruption to conversation or meeting flow.
• Transparency and Interpretability: Each correction is explicitly mapped from user intent to system action, and summaries offered to users emphasize clarity over verbosity.

These design choices enforce a paradigm in which the LLM acts as an "agent facilitator": users are not responsible for traversing system internals, instead making natural interventions that are automatically and interpretably consumed (He et al., 22 Sep 2025).

6. Synthesis and Generalization Across Mediation Contexts

The core paradigm—LLM as interactive mediation agent, with user correction forming an adaptive loop—is generalizable:

• **Real-time feedback harnesses both LLM cognitive throughput (summarization, error localization) and human domain expertise (error detection and disambiguation) to drive continual workflow stabilization.
• **Proactive input structuring (SWM, summarization) ensures both machine and human correctors receive maximally actionable data surfaces, amplifying user impact without overwhelming cognitive load.
• **Mediation feedback not only corrects specific instances but—via online adaptation—steers the system away from recurring error attractors, embodying incremental learning in operational AI deployments.

This operational pattern has broad applicability in settings where automated system outputs are error-prone, high-frequency human engagement is infeasible, and the cost of misattribution or misalignment is high.

7. Future Directions and Open Challenges

Several avenues amplify or challenge current practice:

• Extension to other domains: Analogous mediation loops can regulate code generation drift (Dranias et al., 31 Mar 2026), evaluate peer review (Yun et al., 14 Jan 2026), or steer ODR mediation (Tan et al., 2024), provided feedback channels and prompt structures are context-appropriate.
• Human workload scaling: Quantitative findings highlight the importance of minimizing required interventions—via automatic pre-filtering, summary display, and targeted attention mechanisms.
• Non-stationary model/user dynamics: Ongoing adaptation requires continual system monitoring, periodic prompt revision, and potentially active learning from confirmed corrections.
• Robustness and transparency: Ensuring LLM correction responses are both factually and procedurally aligned with user intentions, especially under ambiguous corrections or adversarial input.

Human-assisted LLM mediation thus offers a blueprint for operationalizing scalable, user-optimized, real-time AI assistants, underpinned by formal error quantification and continual, low-friction human feedback (He et al., 22 Sep 2025).