CritLens: Visual Analytics for Criteria Discovery in Review-Based Decision Making

Abstract: We present CritLens, a visual analytics system that helps users build personalized multi-criteria decision models from review text. In everyday decisions -- choosing equipment, hotels, or restaurants -- evaluation criteria are either preset by platforms or generated by LLMs, leaving users unable to discover, adjust, or verify them against the underlying evidence. This is problematic because many preferences are latent: they surface only upon encountering specific reviews, and any fixed framework risks overlooking low-frequency but decisive details. CritLens addresses this gap by using LLMs to transform reviews into an initial AHP decision model, then supporting iterative, human-in-the-loop refinement. Through coverage gap detection in the embedding space, users discover criteria missed by the initial model; through interactive weight adjustment under AHP consistency constraints, they express personal priorities; and through a multi-level scorecard and exportable decision report, they trace every ranking back to the original review text. Two case studies, an eight-participant user study, and a quantitative consistency-repair experiment demonstrate the system's effectiveness.

• The paper introduces CritLens, a visual analytics framework that iteratively discovers and refines evaluation criteria in multi-criteria decision making.
• The paper presents a novel integration of LLM-generated AHP tree priors with evidence-driven segmentation and visual diagnostics for improved traceability.
• The paper demonstrates effective automated consistency repair, transforming LLM-generated matrices from 57.3% to 100% compliance while enhancing decision justifiability.

CritLens: Visual Analytics for Human-in-the-Loop Criteria Discovery in Review-Based Decision Making

Motivation and Problem Formulation

CritLens addresses a central gap in review-based multi-criteria decision making (MCDM): current frameworks either rely on static, platform-prescribed evaluation criteria or black-box LLM outputs, both constraining users from adaptive criteria discovery and evidence inspection. Many preferences are latent and only emerge upon reviewing pertinent evidence; preset frameworks commonly miss low-frequency but decisive dimensions. Moreover, traceability from score to supporting review is essential for decision justifiability. CritLens elevates MCDM with an LLM-powered, human-in-the-loop approach where users iteratively discover, refine, and verify the evaluative criteria framework, leveraging visual analytics throughout.

System Architecture and Computational Workflow

CritLens comprises four stages: data processing, modeling iteration, decision output, and cross-stage visual analytics support. The architecture operationalizes evidence segmentation and semantic encoding, prior AHP model generation via LLMs, iterative loop for criteria discovery and weight adjustment, and comprehensive traceability with decision report generation.

Figure 1: CritLens system architecture, showing data ingestion, semantic embedding, LLM-based prior generation, iterative visual analytics-assisted refinement, and evidence-backed report output.

The user specifies a decision objective; review text is segmented and encoded in a shared semantic space (using BGE-m3). The LLM generates an initial AHP tree, criteria hierarchy, and pairwise comparison matrices, forming the model's prior. Evidence is aligned and coverage analyzed in embedding space; visual analytics views allow the user to discover coverage gaps, define new criteria with cluster-driven exploration, and adjust weights under AHP consistency constraints. Final scoring and ranking are generated via evidence aggregation and weight composition, with an exportable decision report.

Criteria Discovery and Evidence Coverage

CritLens identifies regions in the embedding space not explained by existing criteria and exposes them visually. Semantic clusters with TF-IDF keyword labels facilitate diagnosis of coverage gaps; density-based clustering (HDBSCAN on UMAP projection) and novelty scores for noise points drive the data-driven exploration of latent dimensions. Uncovered regions are mapped to candidate criteria by submitting their semantics and statistics to the LLM, which proposes integration into the hierarchy.

Figure 2: Case study workflow demonstrating exploration mode for uncovering printhead-related and mechanical jam-related regions, beeswarm evidence visualization, and iterative score evolution.

Coverage is quantified by the proportion of snippets mapped to criteria versus those deemed uncovered. This enables progressive model completion, as users graft new, evidence-backed criteria and delete hallucinated or zero-evidence dimensions.

Weight Adjustment and Consistency Diagnosis

Interactive weight adjustment in CritLens is subjected to AHP mathematical consistency constraints ($\mathrm{CR} \leq 0.1$). Target weights are translated to reciprocal Saaty-scale matrices and, if necessary, repaired via discrete local search to minimize log-distance from user intent while restoring consistency. CritLens provides two modes: automatic matrix adaptation for drag-based adjustment, and visual diagnosis via joint MDS embedding of comparison matrices.

Figure 3: Consistency Diagnosis View, visualizing the spread of projection points across criteria perspectives; primary inconsistency sources manifest as criterion-specific spread.

The joint MDS view reveals inconsistency as spatial dispersion of criterion points across comparison perspectives. Diagnostic overlays expose direct and indirect comparison paths, with interactive repair mechanisms yielding smooth feedback and convergence on mathematical consistency.

Visual Analytics Views for Evidence Traceability

CritLens implements comprehensive juxtaposition-based evidence visualization. The Scorecard View encodes ranking, criterion scores, and evidence distributions (via interactive beeswarm plots), enabling users to confirm the evidence chain for each alternative. Beeswarm plots preserve snippet-level granularity, enabling verification and cross-view mapping. The Quality Index Panel monitors iteration progress—coverage evolution and ranking stabilization.

Figure 4: Case study illustrating criteria deletion, weight adjustment with CR feedback, and evidence-driven score gap analysis for top alternatives.

LLM Analysis Panel synthesizes thematic and statistical summaries of selected clusters, candidate criteria suggestions, and evidence excerpts. The Decision Report provides structured rankings, weight structures, radar comparisons, and review-backed evidence for defensible reporting.

Evaluation: Case Studies, User Study, Consistency Repair

CritLens was validated across two real-world case studies—office printer procurement (Amazon dataset) and Paris hotel selection (HotelRec dataset)—spanning practical scenarios where evidence-driven, adaptive criteria frameworks were essential. Iterative workflow enabled expansion and pruning of criteria based on uncovered evidence regions, substantial coverage gains (from ~59% to ~69%), and enhanced discriminability in rankings. Consistency diagnosis and repair mechanisms proved vital for ranking convergence and model sufficiency feedback.

Figure 5: User study results, showing Likert-scale distributions for criteria discovery, weight adjustment, evidence traceability, and overall usability evaluation.

An eight-participant user study confirmed effective criteria discovery ($\mu=4.13$), evidence traceability ($\mu=4.75$), and strong willingness for adoption in actual decision tasks. Consistency diagnosis visualizations were rated highly by AHP-aware users, though they revealed a background threshold for interpretability. The automated consistency repair experiment demonstrated that only 57.3% of LLM-generated matrices were initially CR-compliant, but repair yielded 100% compliance, with mean CR dropping from 0.105 to 0.007.

Figure 6: Comparison of CR distributions, demonstrating elimination of inconsistency post-repair—the red dashed line marks the CR threshold.

Practical and Theoretical Implications

CritLens establishes new boundaries for evidence-driven, explainable MCDM in review-rich domains. The integration of LLM-powered prior modeling, embedding-based coverage analytics, and visual analytics mechanisms enables transparent, iterative model refinement—critical for scenarios requiring justification, auditability, and collaborative negotiation. The orthogonal separation of evidence aggregation and weight adjustment reduces iteration cost and cognitive load. Practically, CritLens serves procurement officers, administrative teams, and professional consumers in domains where long-tail criteria and evidence traceability are decisive. Theoretically, it demonstrates an operational bridge between document-data-driven LLM outputs and mathematically rigorous human-in-the-loop AHP structures.

Limitations and Future Directions

Current limitations include reliance on snippet rating averages for criterion-level scoring, which may suffer from data sparsity. The hierarchical independence assumption in AHP excludes feedback and inter-criteria dependencies; extension to network-based models (ANP) is indicated. The system omits spatial components useful for geo-sensitive decisions. Further research may address automated convergence metrics, multi-user collaborative workflows, and generalization to non-review text sources.

Conclusion

CritLens pioneers a visual analytics paradigm where criteria discovery, not mere manipulation, is integrated throughout the human-in-the-loop MCDM workflow. LLM-generated models are a baseline; iterative coverage-driven analytics, evidence-backed expansion/pruning, and robust mathematical consistency assure defensible, personalized decision outputs. Case studies, user studies, and quantitative experiments affirm efficacy, with strong implications for auditability and transparent decision support in AI-powered systems.