The Digital Apprentice: A Framework for Human-Directed Agentic AI Development

Abstract: Agentic AI deployments face a recurring design tension: heavy human oversight limits scale, while broad autonomy outruns accountability. Neither posture provides the governance infrastructure required for responsible delegation. We present the Digital Apprentice, a framework for scalable, safe AI agency in which autonomy is earned, not assumed. The Digital Apprentice is a developmental learner that internalizes the tacit methodology of a directing human, graduating through per-skill autonomy tiers only when empirical evidence justifies it. The result is an agent that becomes genuinely useful over time while remaining aligned to a specific human's standards. Three architectural components make this possible. (1) Methodology capture, distilling a directing professional's tacit approach into structured assets. (2) Authorization, with autonomy escalation gated by explicit human approval. (3) Continuous alignment, correcting drift at runtime and converting each correction into owned preference data. We instantiate this framework as an inference-time control plane. We mathematically model the quality framework and discuss policies and techniques designed to raise quality. We apply the framework to an open professional corpus, and we show how catching data drift and applying a different technique at runtime recovers degraded quality dimensions under traffic shift. The implication extends beyond any single application. We believe these three pillars, stitched together as a system, form a safer and more viable path to agentic systems that can scale without sacrificing trust.

• The paper introduces the Digital Apprentice framework, which grants autonomy per skill only after competence evidence and explicit human authorization, while automatically demoting skills after excessive corrections or uncertainty.
• The paper presents ADAPT, an inference-time control plane that branches, scores, triages, and records agent outputs across six quality dimensions while converting human corrections into tenant-isolated preference data.
• The paper’s pilot found that structured onboarding raised mean rubric scores from 0.803 with diversity-gated fusion to 0.957, while runtime recalibration restored actionability from 0.770 to 0.905 under topic drift, though results lacked human-ground-truth validation and statistical testing.

Overview

"The Digital Apprentice: A Framework for Human-Directed Agentic AI Development" (Weber and Taneja, Pheo Inc.) addresses a governance problem in agentic AI deployment: heavy human oversight does not scale, while broad autonomy outruns accountability. The paper contributes two artifacts. The first is a conceptual framework, the Digital Apprentice, in which an agent's authority over each skill is earned through demonstrated competence and explicit human authorization rather than assumed at deployment. The second is ADAPT (Adaptive Data Augmentation and Preference Tuning), an inference-time control plane that operationalizes the framework by running multiple inference policies, scoring outputs along named quality dimensions, detecting drift, switching policies at runtime, and converting every human correction into tenant-isolated preference data. The authors' central claim is that the transition machinery between autonomy levels—what evidence justifies escalation, who authorizes it, and how autonomy is withdrawn—is the missing piece relative to existing autonomy taxonomies.

Graduated autonomy as a per-skill state machine

Autonomy is modeled as a finite state machine indexed per skill, with four tiers: Pre-L0 (observe-only ingestion of human–human data), L0 (sandboxed drafts reviewed before any effect), L1 (candidate outputs that take effect only after human endorsement), and L2 (independent execution under continuous monitoring). Two design choices distinguish this from prior taxonomies of agent autonomy [(Feng et al., 14 Jun 2025); Beer et al. 2014]. First, promotion requires both empirical competence evidence and a recorded human authorization event ($H_{\mathrm{auth}}$); the agent cannot promote itself. Second, demotion is asymmetric and automatic: a skill is rolled back when its correction rate exceeds a threshold or out-of-distribution uncertainty exceeds $u_{\max}$, without waiting for human action. Re-escalation after demotion requires satisfying the graduation criteria again with fresh authorization.

Graduation from tier $L_i$ to $L_{i+1}$ over non-overlapping evaluation windows of $N$ outputs requires three conditions plus authorization: (C1) net improvement, i.e., the current window's correction rate $\rho(W_t)$ is below that of $k=3$ windows earlier; (C2) low residual correction rate $\rho(W_t) \le \tau_{\mathrm{corr}}$; and (Q1) a scorer gate requiring a proportion $p$ of outputs to clear an acceptability threshold $\tau$ on a validated quality scorer $u_{\max}$0. The authors explicitly flag a vulnerability in correction-rate gating: a low correction rate may reflect genuine competence or a reviewer who has stopped checking carefully—the automation-complacency failure documented in human-factors research [Parasuraman & Manzey 2010]. Their response is to treat reviewer engagement as a monitored quantity, recommending periodic seeded checks, though the paper does not specify how engagement is measured operationally.

Learning proceeds in two phases differing in speed and reversibility. Phase 1 uses human corrections as preference pairs retrieved at inference time to steer outputs immediately—a reversible safety buffer requiring no generator update. Phase 2 exports accumulated pairs to model updating (SFT or DPO [(Hoffmann et al., 2022)-style methods; Ouyang et al. 2022; Rafailov et al. 2023]) once volume and statistical significance thresholds are crossed. Notably, the paper concedes that determining domain-specific thresholds for Phase 2 activation is left to future work. Every model update remains traceable to a specific human correction or observed decision, which the authors position as the property distinguishing this loop from open-ended self-improvement.

ADAPT: the inference-time control plane

ADAPT sits between an organization's orchestration layer and its model providers and closes a four-component loop: asset synthesis (methodology, style, and authority exemplars validated by the professional), multi-policy inference (RAG, methodology-conditioned generation, best-of-$u_{\max}$1, diversity-gated fusion), quality telemetry, and preference emission. Each inference event follows a branch–score–triage–emit pipeline. A branch policy produces candidate outputs $u_{\max}$2; each candidate receives a six-dimensional "radar" score vector covering Methodology Fit, Voice/Style Fit, Grounding, Actionability, Context Sensitivity, and Safety Boundary; a pluggable triage scorer (LLM-as-judge rubric, trained reward model, or embedding centroid over approved exemplars) ranks candidates; and every comparison—automatic judge-ranked rejections at weight $u_{\max}$3, human corrections at weight $u_{\max}$4—is emitted as a preference tuple retained as decision memory within the tenant boundary. The framing here is that inference becomes a record-generating event, externalizing each judgment for in-context steering or later model updating.

Drift handling is multidimensional. The control plane distinguishes three causes of degradation—human methodology evolved, agent regressed, or evaluation criteria shifted—each requiring a different response (accelerate observation, roll back and increase review frequency, or revise rubric definitions and re-baseline telemetry). For policy composition, the paper introduces diversity-gated fusion: rather than mutual information between candidates, it computes dispersion in quality-score space,

$u_{\max}$5

over a shortlist $u_{\max}$6 of top-$u_{\max}$7 candidates generated under different framings. Low $u_{\max}$8 indicates candidates collapse onto the same quality profile and fusion is skipped; high $u_{\max}$9 justifies fusing complementary profiles via an operator $L_i$0 gated by threshold $L_i$1. Because radar dimensions are normalized to $L_i$2, $L_i$3 is bounded by $L_i$4 for $L_i$5. The metric's practical appeal is that it reuses scores already computed by the control plane and requires no additional embedding model; the threshold $L_i$6 must be calibrated on validation traffic, which the paper treats as tunable rather than derived.

Proof-of-concept results

The pilot instantiates ADAPT on an open professional-methodology corpus using a Qwen generator and a Gemma LLM-as-judge over the six-dimensional rubric, with 40–60 prompts per arm. Arm A measures onboarding quality; Arm B measures runtime drift recovery under a topic shift.

Condition	Mean triage-stage rubric score
Corpus-only RAG	0.717
Corpus-only best-of-$L_i$7	0.780
Corpus-only diversity-gated fusion	0.803
Structured onboarding-guided policy	0.957
Onboarding policy under topic drift	0.930 (actionability 0.770, safety boundary 0.870)
After diversity-gated recalibration	0.957 (actionability restored to 0.905)

Two findings stand out. First, structured onboarding—the conversion of raw professional artifacts into methodology and style assets—yields a substantially larger gain (+0.154 mean) than any inference-time sampling technique alone, indicating that methodology capture dominates branch-and-triage in this setting. Second, under traffic shift, the onboarding policy degrades selectively: methodology, voice, and grounding hold while operational dimensions (actionability, context sensitivity, safety boundary) deform. Runtime policy switching to diversity-gated fusion recovers actionability from 0.770 to 0.905 and restores the mean to 0.957, supporting the paper's claim that professional output quality is a runtime variable that can be measured, monitored, and repaired without retraining.

The authors are explicit about the evidentiary limits: all reported figures are triage-stage, judge-measured telemetry on a single corpus, not post-human ground truth; the pilot reports no inter-rater agreement, confidence intervals, or significance testing; and the sample sizes are small. The results therefore illustrate the measurement-and-switching loop rather than establishing general effectiveness.

Limitations and risks

The paper identifies several substantive constraints. Methodology capture is framed as an underdetermined inverse problem: following Polanyi's account of tacit knowledge and Suchman's situated-action critique, many world models are consistent with any finite observation set, so recovering a professional's methodology from behavior is ill-posed. The current work restricts scope to text-mediated interaction as a proxy for professional judgment—an assumption that may not extend to multimodal or embodied practice. Consent and confidentiality of recorded human-to-human interaction are flagged as design preconditions, with obligations arising under the EU AI Act and proposed civil-liability rules; lawful basis must be established before the Pre-L0 observation stage begins. Trust dynamics remain a concern: because promotion reads low correction rate as competence, complacency risk persists unless reviewer engagement is actively monitored, and the mechanism for doing so (beyond periodic seeded checks) is not specified. Open technical questions include principled Phase 2 activation thresholds, calibration of the fusion threshold $L_i$8, and validation of the graduation gates against adversarial or degenerate reviewer behavior.

Conclusion

The paper's contribution is an integration pattern rather than a single algorithm: observational learning grounds capability in demonstrated practice, graduation gates and authorization lineage govern escalation and reversal, multidimensional telemetry enables runtime policy switching under drift, and every human judgment accumulates as organization-owned preference data. The proof-of-concept shows the loop functioning end-to-end on one corpus, with strong headline numbers that remain unvalidated by statistical testing or human ground truth. Whether the framework's governance guarantees hold at production scale—and whether tacit-methodology capture generalizes beyond constrained text-mediated domains—are the questions the paper leaves open.