Project Kaleidoscope: Contextual, Human-Aligned Evaluation for Real-World AI Applications

Abstract: Evaluations (Evals) are a deployment bottleneck for real-world AI applications: public benchmarks rarely match a team's users, context, or policies, and human review is often tedious to scale. Motivated by our work with AI applications in the public sector, this project addresses recurring evaluation challenges encountered when applications must satisfy local policy and governance requirements. We present Kaleidoscope, an integrated workflow for contextual functional evaluation that links persona-based test generation, contextualized rubrics, and human review for reliability-gated automated scoring. Generated test cases are scored against application-specific rubrics; human annotations provide reviewable labels; and LLM judges automate scoring only when their agreement with those labels meets a configured threshold. Kaleidoscope is therefore a practical, inspectable, iterative workflow for product teams. We report early evidence from a three-week pilot across four organizational use cases and custom-rubric judge experiments on 108 annotated Q&A pairs spanning four domains and 14 evaluation dimensions. The results highlight useful features for end-to-end reliable, automated scoring.

• The paper introduces an inspectable workflow that combines persona-based test generation, configurable rubrics, human calibration, and reliability-gated LLM judging for application-specific evaluation.
• Experiments on 108 annotated question–answer pairs show that multi-metric judge prompts often reduce human agreement, while single-metric prompts offer modest gains and prompt augmentation produces mixed results.
• A pilot across four enterprise use cases found that 83% of respondents viewed evaluation as more efficient, while highlighting unresolved challenges involving calibration cost, annotation reliability, threshold selection, and agentic workflows.

Kaleidoscope is an end-to-end workflow for contextual functional evaluation of deployed LLM applications, developed at GovTech Singapore. Rather than proposing a new benchmark or a universal judge model, the system integrates persona-based test generation, configurable rubrics, human review, and reliability-gated LLM-as-a-jury scoring into a single inspectable pipeline intended for product teams that must satisfy local policy and governance requirements (2607.14673).

Motivation and problem framing

The paper distinguishes functional evaluation — task-specific assessment of whether a response satisfies user and policy requirements for a given deployment — from general capability benchmarking and safety assessment. The authors argue that public benchmarks rarely match an organization's users, context, or policies, and that existing tooling (OpenAI Evals, Inspect AI, Promptfoo, EvalAssist) covers parts of the eval lifecycle but leaves open the integration of contextual test construction, customizable rubrics, human calibration, and gated automated judging into one workflow. This is consistent with prior findings that under-specified evaluation contexts force arbitrary judgments and produce inconsistent conclusions about model quality.

Four design considerations operationalize the workflow: representative test cases (R1), configurable reviewable rubrics (R2), a human-in-the-loop experience that minimizes reviewer fatigue (R3), and transparent, reliability-gated automated scoring (R4).

Workflow design

The workflow separates three artifacts: the evaluation set, human labels for calibration, and post-calibration automated scores. Test generation diversifies questions along five dimensions — persona, typical/edge type, in/out knowledge-base scope, input style, and language — with default allocation ratios tuned after pilot feedback to weight realistic usage (70% typical + in-KB when a KB exists). Personas can be user-defined, generated from the application profile, or sampled from open-source persona datasets.

Human review is LLM-assisted: a first-pass judge proposes labels and highlights claims, shifting reviewers from one-shot labeling toward auditing. To mitigate automation bias, reviewers must select from rubric options rather than approve/reject judge suggestions. The annotation unit follows the rubric: claim-level rubrics label extracted claims; response-level rubrics label whole responses.

For automated scoring, Kaleidoscope instantiates three candidate judges per rubric and measures each against the human-labeled calibration subset. Only judges passing a local gate of Macro F1 > 0.5 contribute to majority-vote aggregation; if none passes, the aggregate is withheld and flagged for human review. The authors are explicit that this gate is a local quality-control check against human labels, not evidence of universal judge correctness. Claim-level accuracy scoring uses deterministic sentence tokenization, filtering plus LLM-based checkworthiness classification, per-claim binary verdicts against the knowledge base, and strict aggregation where any unsupported claim renders the response inaccurate.

Rubric and judge experiments

The design was informed by formative experiments on 108 annotated question–answer pairs spanning four organizational AI systems (factual and subjective domains), 14 evaluation dimensions, two annotators, and eight candidate judge models (GPT-4.1, GPT-5-mini, Claude Haiku 4.5, Grok 4.1 Fast, Gemini 2.5/3/3.1 Flash Lite, GLM-5). Judges were evaluated separately against each annotator and metrics averaged, rather than using consensus labels.

Multi-metric brackets fail. Grouping related metrics into single judge calls ("Relevancy" and "Voice" brackets) yielded low-to-moderate agreement: best bracket-level Cohen's $\kappa$ of 0.239 (Relevancy, GLM-5) and 0.376 (Voice, Claude Haiku 4.5). Per-metric analysis revealed wide variance hidden by aggregation — bias reached $\kappa = 0.818$ while coherence fell to $\kappa = 0.174$, with no single model consistently strongest.

Prompt iteration does not rescue brackets. Three rounds of prompt refinement degraded Relevancy ($\kappa$ from 0.239 to 0.192, −19.5%) and left Voice flat ($\kappa$ 0.376 → 0.373). The authors attribute this partly to scope overload but concede that rubric ambiguity and class distribution may also contribute.

Single-metric judges help modestly. Dedicated prompts improved empathy ($\kappa$ 0.485 → 0.559, +15.2%) and verbosity ($\kappa$ 0.337 → 0.376, +11.6%) over the multi-metric baseline, though final $\kappa$ values remained moderate. The comparison confounds prompt scope with model selection, so gains cannot be attributed solely to narrowing scope.

LLM prompt augmentation is mixed. Using four augmentation models over three rubric-detail levels with a fixed downstream judge, structure improved ($\kappa$ 0.514 → 0.548) but specificity regressed ($\kappa$ 0.366 → 0.336). Notably, more detailed rubric inputs did not consistently yield better augmented prompts, and the weak specificity result coincides with low inter-annotator agreement on that metric ($\kappa = 0.818$0 vs. 0.666 for structure), suggesting ambiguous human decision boundaries limit achievable judge alignment.

The cumulative design implication is a hybrid architecture: validated preset rubrics where available, one-metric-per-judge-prompt for custom criteria, and LLM augmentation as a scalable starting point subject to human review and iterative refinement.

Pilot with business users

A three-week pilot across four use cases (finance claims bot, HR bot, procurement bot, staff assistant) involved eight testers who ran approximately 12 evaluation runs, 180 generated test cases, and 40 human-reviewed outputs. Six respondents completed the questionnaire: 83% reported more efficient evaluation, and all reported consulting the judge reliability score when interpreting results. The pilot was explicitly not a controlled comparison against manual evaluation or existing tools, nor a validation of correctness.

Qualitative feedback drove concrete revisions: rebalanced test allocation away from uniform edge-case coverage, added web-search grounding for sparse application profiles, retained multi-judge defaults despite low user adoption (attributed to unfamiliarity), and added import/export points so teams could integrate partial workflows rather than restructure around the tool.

Limitations and open questions

The paper is candid about constraints. The calibration experiments are formative, conducted on a small shared development dataset, and cannot support claims about model rankings generally. The pilot's scale, duration, and first-time-user population limit generalizability, and some friction likely reflects limited evaluator familiarity rather than workflow defects. Several structural issues remain open:

• Calibration at scale: frequent evaluations make large review sets impractical, risking skipped calibration and unguarded reliance on judge scores.
• Annotation robustness: the current process uses single annotators; multi-annotator aggregation would capture rater variation but increases manual cost.
• Cost: claim-level and multi-judge scoring multiply model calls substantially, requiring explicit cost controls.
• Scope: the workflow evaluates input–output behavior only, excluding agentic traces, retrieval diagnostics, and multi-turn dynamics.
• Threshold selection: neither the Macro F1 gate nor pass/fail score thresholds have principled defaults; both depend on application risk level.

Conclusion

Kaleidoscope contributes a practical, inspectable workflow demonstrating that contextual functional evaluation requires calibrated integration of test construction, application-specific rubrics, efficient human review, and reliability-gated judging. Its strongest empirical signals are negative and diagnostic: multi-metric judge prompts degrade alignment, prompt iteration alone does not recover it, and augmentation effectiveness is criterion-dependent. The evidence base — a 108-example formative study and a six-respondent pilot — supports the design choices as reasonable but leaves validation of long-term validity, calibration economics, and extension to agentic and retrieval-augmented systems as clearly stated open problems.