Quadrant Evaluation Framework

• Quadrant Evaluation Framework is a methodology that maps entities into four distinct quadrants based on two independent, domain-specific axes.
• It employs normalized scores and rigorously defined thresholds for metrics like smartness and automation to enable comparative evaluation.
• The framework is applied in business AI, clinical method comparisons, and LLM persona taxonomies, offering clear visualization and actionable insights.

A Quadrant Evaluation Framework is a structured analytic methodology that uses a two-dimensional space to comparatively position entities, attributes, or measurement trends according to two semantically independent axes. Quadrant frameworks have been deployed in diverse technical contexts, ranging from business AI “intelligence” and automation (BenBassat, 2018), trending-agreement in method-comparison studies (Hiraishi et al., 2020), to systematizing the design and assessment of LLM personas (Sun et al., 4 Nov 2025). The organizing principle is that by assigning a pair of coordinates derived from domain-specific axes, artifacts or systems can be mapped into one of four mutually exclusive quadrants, each corresponding to a qualitatively distinct class or behavioral regime.

1. Mathematical Foundations and Axis Construction

Quadrant frameworks rely on rigorous axis definitions and normalization. Each axis is constructed to be interpretable, reproducible, and—where possible—normalized onto a unit interval or binary domain. In the “AIQ Quadrant” for business AI software (BenBassat, 2018), two composite metrics are computed:

• Smartness ($S_i$): A weighted sum of normalized KPIs relevant to decision quality. For $n$ KPIs with weights $\{w_k\}$ ($\sum_{k=1}^n w_k = 1$) and normalized metric $M_{i,k}$ for system $i$, $S_i = \sum_{k=1}^n w_k M_{i,k}$.
• Automation Level ($A_i$): The mean degree of automation across $m$ micro-tasks, where $a_{i,j} \in \{0, 0.5, 1\}$ for manual, partially assisted, or fully automated.

Continuous axes are discretized at representative thresholds (e.g., $0.5$), splitting the space into quadrants by low/high regime for each axis. In other quadrant frameworks, such as the Four-Quadrant Technical Taxonomy for LLM-Based Personas (Sun et al., 4 Nov 2025), axes are categorical (Deployment Modality $D$: Virtual=0, Embodied=1; Interaction Intent $C$: Emotional=1, Functional=0).

2. Quadrant Definitions and Interpretation

Systematic labeling and interpretation of the four quadrants is intrinsic to the framework:

Quadrant	Axis 1: Low	Axis 1: High
Low	I	II
High	III	IV

In the AIQ Quadrant:

• QI: Borderline AI—low smartness, low automation.
• QII: Smart but manual—high smartness, low automation.
• QIII: Automated but dumb—low smartness, high automation.
• QIV: True Business AI—high smartness, high automation, with further “A”, “AA”, “AAA” levels reflecting added ML or digital assistants (BenBassat, 2018).

In the LLM persona taxonomy:

• Quadrant I: Virtual Emotional Companion,
• Quadrant II: Virtual Functional Assistant,
• Quadrant III: Embodied Emotional Companion (general),
• Quadrant IV: Embodied Functional Assistant (vertical domains) (Sun et al., 4 Nov 2025).

For trending-agreement plots, quadrant assignments reflect concordant/discordant trend changes between methods (Hiraishi et al., 2020).

3. Scoring, Thresholds, and Algorithmic Implementation

The quadrant assignment depends on a precise scoring and normalization process. For composite metric axes, normalization is essential for comparability:

Example: AIQ Quadrant Scoring (Editor’s term: “(S,A) vector”)

• Normalize each KPI to $[0,1]$:

$$M_{i,k} = \begin{cases} (x_{i,k} - x^{min}_k) / (x^{max}_k - x^{min}_k), & \text{higher-is-better} \ (x^{max}_k - x_{i,k}) / (x^{max}_k - x^{min}_k), & \text{lower-is-better} \end{cases}$$

• Aggregate weighted KPIs: $S_i = \sum_{k} w_k M_{i,k}$.
• Calculate automation: $A_i = \frac{1}{m} \sum_{j=1}^m a_{i,j}$.
• Quadrant decision: compare $(S_i,A_i)$ to $(0.5,0.5)$ threshold.

For repeat-measurement agreement (Hiraishi et al., 2020), quadrants categorize joint trends in $(\Delta x, \Delta y)$, with an exclusion zone $Ez(a)$ to filter noise:

• Define concordant pairs: Quadrant I (NE, both increase), Quadrant III (SW, both decrease).
• Compute classical and per-individual concordance rates post-exclusion, with the option to set a minimum-concordance threshold $m$ for per-subject robustness.

4. Applications: Case Studies in Business AI, Method Agreement, and Persona Taxonomy

Quadrant frameworks have been adapted to distinct technical domains:

Business AI (AIQ Quadrant)

Deployment of field-service scheduling solutions illustrates quadrant allocation:

• Systems with $(S,A)\approx(0.1,0.1)$ and $(0.25,0.15)$ are in QI.
• A full optimizer with $(0.82,0.85)$ lies deep in QIV.
• ML enhancements and digital assistants (e.g., Google Glass–enabled Butler) further elevate solutions to AAA status through improved $S$ and $A$ (BenBassat, 2018).

Trending-Agreement in Clinical Methods

The four-quadrant plot is employed to evaluate directional agreement between new and gold-standard clinical measurements, with robust estimation via per-subject concordance thresholds and ROC/AUC benchmarking for diagnostic accuracy (Hiraishi et al., 2020).

AI Persona Design Taxonomy

The four-quadrant technical taxonomy articulates a systematic map for LLM-based persona applications, guiding technical requirements, evaluation, and risk assessment in each quadrant via domain-specific metrics and layered frameworks (Sun et al., 4 Nov 2025).

5. Evaluation Metrics, Visualization, and Interpretation

Each quadrant framework specifies, where relevant, metrics and visualization conventions:

• Plotting $(A_i,S_i)$ pairs enables rapid visual comparison of business AI alternatives.
• In trending-agreement, four-quadrant scatterplots are annotated with concordant rates, and ROC/AUC provide discriminative power assessment (Hiraishi et al., 2020).
• LLM persona taxonomies recommend radar (spider) plots for multi-layered capability comparison, with the (D,C) pair mapped to its quadrant (Sun et al., 4 Nov 2025).

Metric formulas are always domain-adapted: e.g., Persona Consistency Rate (character hallucination incidence), Task Success Rate (fraction of completed tasks), Symbol Grounding Accuracy, Clinical Efficacy Score, Safe-Response Rate. No generic scoring formula is universal across all quadrant frameworks.

6. Best Practices and Limitations

Critical methodological considerations include:

• Axes and scoring procedures must be co-designed with domain stakeholders to ensure business/clinical relevance (BenBassat, 2018).
• Normalization thresholds must be continuously validated against evolving baselines.
• Quadrant frameworks are inherently comparative; their primary function is relative rather than absolute evaluation (“not an absolute intelligence meter” (BenBassat, 2018)).
• Per-individual concordance rates and exclusion thresholds yield better robustness and interpretability in longitudinal method-comparison contexts (Hiraishi et al., 2020).
• Qualitative attributes (trust, explainability, regulatory compliance) are typically external to core quadrant metrics but may be incorporated as additional layers in multi-layered frameworks (Sun et al., 4 Nov 2025).

Limitations persist: subjectivity in KPIs or micro-task decomposition, overweighting of automation potentially incentivizing less transparent systems, and difficulty in capturing emergent user-centric factors.

7. Synthesis and Broader Impact

Quadrant Evaluation Frameworks have become foundational tools in business AI, experimental method comparison, and AI system taxonomy. Their appeal lies in codifying complex multi-criteria evaluations into compact geometric and categorical mappings. This structure aids both technical optimization and stakeholder communication, fosters comparability across solutions, and enables systematic tracking of technological progress (e.g., tracking “A” to “AA” to “AAA” advancements in business AI through incremental metric improvements (BenBassat, 2018)). As technical domains evolve, quadrant frameworks remain adaptable, provided axes reflect the principal sources of value and risk in the target application.

References

• BenBassat, "AIQ: Measuring Intelligence of Business AI Software," (BenBassat, 2018)
• Yamamoto et al., "Concordance Rate of a Four-Quadrant Plot for Repeated Measurements," (Hiraishi et al., 2020)
• Su et al., "Systematizing LLM Persona Design: A Four-Quadrant Technical Taxonomy for AI Companion Applications," (Sun et al., 4 Nov 2025)