More than a Judge: An Empirical Study of Agent-Human Interaction in Crowdsourced Testing Assessment

Abstract: Agentic AI is increasingly being integrated into software engineering workflows. In crowdsourced testing, however, the large volume and uneven quality of submitted reports still create a substantial review burden for developers. In prior work, we developed and validated a multi-agent assessment backbone based on the LLM-as-a-Judge paradigm. That backbone assesses reports along three dimensions--textuality, adequacy, and competitiveness--and was shown to align well with human consensus while substantially reducing assessment effort. Yet reliable automated judging does not by itself show whether agent outputs can improve human work when embedded into workflow. This paper studies that missing question in the context of crowdsourced testing. We investigate whether assessment-derived, actionable feedback can improve how testers revise reports, perform on later tasks, and transfer reporting practices across applications. To do so, we conducted a controlled four-stage human-subject study with 20 testers across three real-world applications. The results show that agent-generated feedback supports immediate improvements in revised reports, better first submissions on a new task after prior feedback exposure, and evidence of partial but meaningful transfer to a later application. A post-task questionnaire completed by 17 participants complements these artifact-based findings by suggesting that the feedback was generally understandable, acted upon in revision, and carried into later tasks, while also revealing remaining friction in specificity and execution. Overall, the study provides empirical evidence that, in the studied crowdsourced testing setting, assessment agents can serve not only as post-hoc judges but also as workflow-integrated feedback providers that support upstream report-quality improvement.

• The paper demonstrates that multi-dimensional agent feedback significantly improves report quality and requirement coverage, with gains up to +27.65% in adequacy.
• It employs a controlled human-subject experiment comparing feedback-integrated and control groups across three application tasks to assess immediate and transferable effects.
• The results imply that embedding LLM-based feedback within testing workflows not only enhances immediate report revisions but also fosters durable skill transfer among testers.

Multi-Dimensional Agent Feedback in Crowdsourced Testing: From Automated Assessment to Workflow Integration

Introduction

Agentic AI systems featuring LLM-based evaluators are increasingly embedded in software engineering workflows, promising to alleviate evaluative bottlenecks without sacrificing reliability. In crowdsourced testing, where the diversity and volume of submitted reports overwhelm developer review bandwidth, LLM agents as assessors have demonstrated strong alignment with human consensus and substantial efficiency improvements. However, such assessments address only the post-hoc scoring problem; the latent challenge is whether agent-generated outputs, when integrated back into the tester workflow as actionable feedback, yield improvements in human artifact quality and knowledge transfer across tasks.

This paper—"More than a Judge: An Empirical Study of Agent-Human Interaction in Crowdsourced Testing Assessment" (2606.06301)—signals an important shift in focus from validating agent judgment to measuring the human impact of workflow-integrated, assessment-derived feedback. Through a controlled human-subject experiment, the study provides empirical measurement of how multi-dimensional agent feedback influences crowdsourced testers’ revision practices, initial performance on new tasks, and skill transfer across application contexts.

Background and Problem Context

Crowdsourced testing leverages heterogeneously skilled contributors to augment test coverage and defect detection. The critical challenge is the uneven quality and scale of report submissions, often with ambiguous structure, missing context, or incomplete requirement coverage, as illustrated by the qualitative contrast between "good" and "bad" reports.

Figure 1: Examples contrasting high- and low-quality crowdsourced test case and defect reports.

Prior automated approaches address filtering, clustering, and deduplication but are generally limited to passive information triage, rarely providing actionable report-level feedback to human testers. The emergence of the LLM-as-a-Judge paradigm supports robust, interpretable, multi-dimensional report scoring—not only scalar reward values—enabling both assessment and rationale generation. However, there is limited empirical evidence that such agent outputs, when recycled into the upstream workflow as structured feedback, realize practical improvements in tester performance or catalyze learning across tasks.

Multi-Dimensional Assessment Backbone

Building on previously validated LLM-based multi-agent assessors, the system architecture operationalizes three orthogonal assessment dimensions:

• Textuality: Evaluates report atomicity, completeness, conciseness, understandability, and reproducibility via a composite indicator checklist. Dual LLM scoring with disagreement arbitration ensures robust, interpretable judgment.
• Adequacy: Measures requirement coverage by mapping test cases to a hierarchy of atomic functional points constructed from task requirement documents, calculating precise functional coverage.
• Competitiveness: Assesses the rarity and uniqueness of discovered defects across the tester cohort via semantic clustering, rewarding novel and non-duplicative findings.

  Figure 2: Systemic workflow showing how multi-dimensional assessment agents integrate as both evaluators and feedback providers.

Textuality and adequacy dimensions are uniquely convertible into actionable feedback: structured revision suggestions for missing report elements and coverage guidance highlighting unaddressed requirement nodes. Competitiveness, dependent on post-hoc aggregation, is reserved for downstream analysis and is not included in immediate feedback cycles.

Human-Subject Study Design

The central experimental question asks whether agent-generated, multi-dimensional feedback embedded into the test workflow leads to measurable and transferable improvements in crowd tester performance. The staged procedure uses two groups (A/B), three applications (APP1/2/3), and a four-phase workflow:

Figure 3: Four-stage human-subject workflow for evaluating in-situ agent-generated feedback and its transference effects across tasks.

1. Baseline (APP1): Establish initial comparability and skill levels.
2. Intervention (Group A on APP1) vs Control (Group B proceeds to APP2, no feedback).
3. New Task/Transfer: Cross-group comparison on APP2, measuring immediate transfer after feedback exposure.
4. Long-term Transfer: Both groups execute APP3, evaluating durable skill propagation.

Twenty testers (with heterogeneous but systematically trained backgrounds) serve as subjects. A post-study questionnaire supplements behavioral artifact analysis with self-reported data on feedback usage, external AI tool influence, and perceived feedback efficacy.

Experimental Results and Analysis

Immediate Feedback Effects (RQ1)

Intervention data show consistent, strong within-task improvements after agent feedback: mean group increases of +11.75% (testcase textuality), +9.15% (defect textuality), +10.46% (composite textuality), and +27.65% (adequacy) for Group A, with Group B mirroring textual improvements but more modest adequacy gains. Most testers shifted from partial to near-complete requirement coverage, with a majority attaining 100% adequacy after revision. Gains are most pronounced for lower-performing testers, indicating headroom-driven improvement.

Transfer to New Task (RQ2)

Feedback-exposed testers (Group A) substantially outperform controls (Group B) in first submissions on an unseen app (APP2), with deltas of +11.44, +5.58, and +8.51 points on testcase, defect, and overall textuality. Adequacy transfer is masked due to domain familiarity ("e-commerce" with high baseline coverage), but after exclusion of a persistent low-coverage outlier, Group A's functional coverage surpasses B. Textual improvements stem from behaviors directly aligned with prior feedback: explicit preconditions, requirement-aligned phrasing, and clearer steps.

Cross-Application Skill Transfer (RQ3)

After completing at least one feedback cycle, both groups exhibit robust improvements on APP3 relative to the original baseline: +7.31 (testcase textuality), +2.64 (defect textuality), +4.98 (overall textuality), and +7.40 (adequacy) on average. Improvement is broad-based across testers, not driven by outliers. Some per-tester regressions are attributable to ceiling effects or task-dependent adaptation, but artifact and self-report evidence confirm durable, transferable learning.

Complementary Questionnaire Results

Analysis of the post-task questionnaire reveals:

Figure 4: Summary of external AI tool usage during tasks; primary uses involve idea organization, requirement comprehension, and report wording.

• Occasional support from external AI tools was common; systematic or dominant usage was rare, suggesting limited confounding impact.
• Most testers found the agent feedback actionable and specific, translating it into concrete revisions and actively adding missing test points and coverage.
• The majority reported deliberate carryover of learned report-writing and coverage habits to subsequent tasks.

  Figure 5: Distribution of responses regarding perception, trust, and adoption of agent-generated feedback.

Feedback was not uniformly frictionless: some found the feedback overly generic or difficult to operationalize, especially for nuanced requirement decomposition.

Figure 6: Thematic summary of what testers found most helpful, confusing, transferable, and needing improvement within the feedback mechanism.

Discussion and Implications

This study empirically demonstrates that previously validated assessment agents provide workflow value beyond judgment when their outputs are recycled as actionable, structured feedback to human testers. Measured improvements are not only strong for immediate revision but exhibit generalization to new tasks and non-trivial skill transfer, refuting any strictly score-alignment explanation. The evidence points to a learning mechanism at two levels: micro (clearer reporting) and macro (systematic, requirement-oriented coverage and planning). However, friction persists at the translation boundary where high-level coverage feedback must be operationalized into concrete actions, especially in the presence of ingrained suboptimal habits.

Practically, this suggests that agentic AI integration should move upstream, positioning automated assessment at report creation points within the workflow to catalyze both immediate artifact improvement and upstream tester development. Theoretical implications include differentiating between agents as static judges and as situated, adaptive workflow participants, and pointing toward further research on continuous, workflow-integrated feedback cycles, adaptivity, and cross-domain transferability.

Conclusion

"More than a Judge" (2606.06301) substantiates the workflow impact of LLM-based assessment agents in crowdsourced testing. Integrated, multi-dimensional feedback drives measurable gains in both report textuality and requirement coverage, with evidence of transfer to both new tasks and new testers. The agentic role thus expands from passive judge to active participant in quality improvement cycles. Future studies are warranted on generalization across artifact types, domain adaptation, and the design of advanced, context-sensitive feedback intervention protocols.