Dynamic Agentic AI Expert Profiler System Architecture for Multidomain Intelligence Modeling

Abstract: In today's artificial intelligence driven world, modern systems communicate with people from diverse backgrounds and skill levels. For human-machine interaction to be meaningful, systems must be aware of context and user expertise. This study proposes an agentic AI profiler that classifies natural language responses into four levels: Novice, Basic, Advanced, and Expert. The system uses a modular layered architecture built on LLaMA v3.1 (8B), with components for text preprocessing, scoring, aggregation, and classification. Evaluation was conducted in two phases: a static phase using pre-recorded transcripts from 82 participants, and a dynamic phase with 402 live interviews conducted by an agentic AI interviewer. In both phases, participant self-ratings were compared with profiler predictions. In the dynamic phase, expertise was assessed after each response rather than at the end of the interview. Across domains, 83% to 97% of profiler evaluations matched participant self-assessments. Remaining differences were due to self-rating bias, unclear responses, and occasional misinterpretation of nuanced expertise by the LLM.

• The paper proposes a modular, LLM-driven profiling system that dynamically classifies expertise with transparent scoring and converges rapidly, achieving up to 98% accuracy.
• The methodology integrates six sequential layers—from input to output—using weighted feature scoring with LLaMA v3.1 for fine-grained linguistic analysis.
• The evaluation demonstrates high agreement (83–98%) with self-assessments in both static and dynamic modes, validating the system’s adaptability and robustness.

Dynamic Agentic AI Expert Profiler System Architecture for Multidomain Intelligence Modeling: An Expert Summary

Introduction

This work introduces a modular architecture for dynamic expertise profiling, leveraging LLMs for fine-grained, interpretable classification of human expertise from natural language responses. The need for such an agentic profiler is motivated by the increasing demand for adaptive, context-aware AI-human interactions spanning domains with substantial inter-participant variability in expertise. Existing expert profiling systems predominantly rely on structured metadata, co-authorship graphs, or static ML models, lacking real-time adaptability and domain-linguistic interpretability. The proposed framework directly addresses this gap with a layered architecture, real-time and offline operation, and transparent scoring methodologies.

Layered System Architecture

The system is structured into six principal layers: Input, Preprocessing, Feature Scoring, Aggregation, Classification, and Output, each with modular subcomponents optimizing expertise determination and explainability.

Figure 1: High-level system architecture showcasing sequential, modular layers for receiving textual responses, preprocessing, linguistic scoring, dynamic aggregation, and explainable user-oriented output.

At input, the architecture supports both transcripts for batch-mode profiling and live response streams for interactive interviews, enabling robust multidomain applicability. The preprocessing layer ensures textual normalization and domain-adaptive lexicon mapping, reducing bias from linguistic irregularities. Segmentation enables chunk-wise parsing for finer granularity in subsequent analysis.

The Feature Scoring Engine is the computational core, with parallelized modules quantifying terminology application, conceptual depth, real-world application, logical rigor, and uncertainty signals. Each module utilizes the LLaMA v3.1 (8B) LLM for zero-shot and few-shot semantic scoring, with normalized outputs that subsequently undergo global penalization for incorrectness or boosting for accurate, evidence-backed assertions.

The Aggregation layer synthesizes the five normalized feature indicators through weighted dimensions of Relevancy (0.5), Recency (0.3), and Consistency (0.2), reflecting empirically tuned importance of application, conceptual currency, and answer stability. This ensures multi-criteria fusion and resilience to overfitting from any single criterion. The Classification layer applies contiguous scoring bands for transparent expertise assignment (Novice, Basic, Advanced, Expert), while the Output layer generates detailed, machine- and human-intelligible reports, supporting JSON export and user-friendly natural language summaries.

Evaluation Protocols and Findings

The system’s efficacy was validated in both static (transcript-based) and dynamic (real-time adaptive interview) modes, using LLaMA v3.1 (8B) for inference. Validation cohorts spanned security/privacy (structured technical) and gamification/awareness (interpretive, context-driven) domains.

Numerical agreement between the profiler and participants’ self-evaluations attained 83–97%. For security, static profiling achieved 98% match with self-assessment; dynamic profiles yielded 97% agreement for both security and LLM awareness. Lower agreement in broad interpretive domains (privacy/gamification, 83–89%) reflects the profiler’s challenges with vague or highly subjective input. Discrepancies primarily resulted from self-assessment biases and sporadic LLM misjudgments regarding nuanced expertise.

In dynamic mode, the real-time profiler adapted questioning difficulty based on its continuously updated expertise estimate. For technical domains, the profiler’s classification typically stabilized within 2–3 questions. By question 2, 100% of security participants' expert status was correctly classified and remained stable—a clear demonstration of the profiler’s rapid convergence in well-defined domains. For privacy or LLM awareness, exact matching was achieved by the third response in all cases. Critically, within one level of accuracy, the profiler’s estimates were correct for almost all participants after the first or second response across domains, suggesting rapid coarse-grained expertise detection.

Interpretability, Objectivity, and Systemic Robustness

A key systemic advantage is transparent, multi-source justification for each final classification. The justification generator traces influential score components, aiding both accountability and post-hoc diagnostic analysis. The system mitigates participant overconfidence/underconfidence by strictly evaluating linguistic and reasoning evidence, decoupled from self-representation.

Penalty/boost mechanisms further ensure that rare instances of factual errors or exceptional explanations have proportional impact, preventing artifactually inflated ratings. Insufficient evidence detection prevents overfitting in undersampled cases.

The system’s explainability, modularity, and rapid adaptation enable use in high-stakes settings where fair, consistent, and evidence-based expertise judgments are required, even across shifting interview contexts or participant populations.

Implications and Future Directions

The profiler’s architecture generalizes well to domains where expertise manifests in natural language but varies in structure, granularity, and epistemic stances. Integration in educational adaptive testing, organizational workforce assessment, and context-aware intelligent tutoring is immediate. Its resilience and domain-adaptive normalization confer robustness against input distributional shifts.

From a theoretical perspective, the architecture embodies a practical pipeline for aligning LLM-based assessment with multidimensional construct operationalization. The weighted, modular dimensions provide an interpretable mapping from latent linguistic competence to actionable assessment bands, supporting principled future work in psychometric AI and automated human-computer interaction research.

Future avenues include dynamic retraining for domain-specific thresholds and weights, benchmarking with larger, multilingual datasets, and exploration of advanced aggregation strategies for deeper context fusion (e.g., attention-based or graph-based aggregation). The unresolved challenges around vagueness and subtle interpretive domains highlight ongoing opportunities for hybrid human-in-the-loop scoring, meta-learning, and uncertainty quantification.

Conclusion

This research describes a technically rigorous, empirically validated, and interpretable agentic LLM-powered expert profiling pipeline. By unifying layered modular design, real-time dynamic adaptation, and transparent multi-criteria aggregation, the system achieves high-accuracy, bias-resistant expertise classification. These features position it as a foundational infrastructure for adaptive, scalable, and fair AI-driven assessment and interactive systems in multidomain settings.