Skill-Augmented AI Agents for Medical Research Analysis: An Exploratory Multi-Model Human Evaluation in an NSCLC Transcriptomic Biomarker Task

Abstract: Background. LLMs and AI agents are increasingly used to support biomedical research, but native model outputs may omit key analytical steps, misuse methods, or overstate conclusions. We evaluated whether autonomous access to a medical research skill package was associated with higher-quality AI-generated transcriptomic research-analysis outputs compared with native AI without skills. Methods. We conducted an exploratory multi-model human evaluation using a non-small cell lung cancer immunotherapy biomarker task. Six model backbones were tested. The evaluation included 21 anonymized outputs: 9 native-AI outputs and 12 skill-augmented outputs generated through an AI agent implementation represented by OpenClaw. Four non-expert biomedical reviewers and two blinded experts evaluated each output, with two ratings from each reviewer type. The primary outcome was expert-rated overall quality. Results. Skill-augmented outputs showed directionally higher expert overall quality than native-AI outputs (mean 5.50 vs 5.11; difference=0.39; bootstrap 95\% CI, -0.04 to 0.90; Welch p=0.156). Non-expert reviewer quality showed the same direction (mean 4.72 vs 4.47; difference=0.26; bootstrap 95\% CI, -0.25 to 0.80; Welch p=0.373). Expert agreement was limited (single-rating ICC=-0.15), and model-specific effects were descriptive and heterogeneous. Conclusions. Autonomous skill access showed a directional quality signal in this exploratory sample, but the signal was smaller than expert-rating noise and should not be interpreted as confirmatory evidence. The findings primarily motivate larger evaluations of skill-augmented AI agents with stronger reliability controls, platform replication, and biological-validity assessment.

• The paper’s primary contribution is demonstrating that augmenting AI agents with medical research skills can directionally improve analysis output quality in NSCLC tasks.
• It employs a multi-model, human-evaluated framework to compare native AI and skill-augmented systems, revealing heterogeneous performance across different LLM architectures.
• Key findings highlight challenges in evaluator agreement and model-specific variability, underscoring the need for improved human evaluation methods in biomedical AI.

Skill-Augmented AI Agents in Biomedical Research Analysis: An In-Depth Evaluation

Introduction

The paper "Skill-Augmented AI Agents for Medical Research Analysis: An Exploratory Multi-Model Human Evaluation in an NSCLC Transcriptomic Biomarker Task" (2606.11830) investigates the impact of medical-research skill augmentation on the quality of AI-generated biomedical research analyses. The study operationalizes this in the context of non-small cell lung cancer (NSCLC) transcriptomic data, exploring whether autonomous access to a procedural skill package enhances output quality compared to standard ("native") LLM-driven approaches. Rigorous human evaluation is employed across multiple state-of-the-art LLM backbones, providing nuanced insights into the benefits and limitations of agentic skill augmentation in real-world biomedical workflows.

Experimental Design and Methodology

The evaluation leverages a multi-model, human-assessed framework comparing two primary conditions: native AI (baseline LLMs without extra procedural skills) and skill-augmented AI (LLMs with autonomous access to a curated medical research skill package via the OpenClaw agent). Six LLM backbones (GPT-5.4, Claude Sonnet 4.6, GLM-5.1, DeepSeek-V4 Pro, Kimi K2.6, MiniMax-M2.7) were included, producing a total of 21 anonymized outputs assessed by four non-expert biomedical reviewers and two blinded domain experts.

Every model was assigned an identical, complex biomedical prompt: constructing a multi-gene signature for NSCLC immunotherapy response prediction, and analyzing the role of programmed cell death (PCD) mechanisms. Outputs were evaluated with respect to clarity, methodological rigor, workflow integration, and perceived risk, using detailed Likert-scale metrics.

Figure 1: Evaluation dataset quality-control flow, documenting anonymized outputs, generation-strategy distribution, human rating counts, and reviewer attention checks.

The medical research skill package covered a broad spectrum of bioinformatics processes, including normalization, cohort design, differential analysis, enrichment and pathway analysis, immune infiltration assays, machine learning construction/validation, and systems-biology modules. Output review was performed using anonymized artifacts to ensure unbiased human ratings, with inter-rater reliability assessed systematically.

Primary Evaluation Results

The principal outcome was expert-rated overall quality. Skill-augmented outputs demonstrated a directionally higher, but not statistically significant, improvement in mean expert score (5.50 vs 5.11 for native AI; Δ=0.39, 95% CI: -0.04 to 0.90; Welch p=0.156). Non-expert reviewer scores echoed this directional trend. Expert methodological quality showed a slightly larger effect (Δ=0.47, 95% CI: 0.12 to 0.95), but again, the confidence intervals overlap with zero, indicating statistical uncertainty.

Figure 2: Expert-rated overall quality and non-expert reviewer quality by generation strategy, demonstrating directional, but not statistically significant, improvements for skill-augmented outputs.

Further, skill-augmented generation did not reliably lower perceived methodological risk according to non-expert ratings. These results should be interpreted against the backdrop of substantial expert disagreement (single-rating ICC=-0.15), with inter-expert variability exceeding the effect size of skill augmentation itself.

Figure 3: Expert methodological quality (left) and non-expert perceived risk (right) by generation strategy, highlighting moderate effect sizes and variable perceived risk.

Model-Specific Heterogeneity and Variability

Model-level analyses reveal considerable heterogeneity:

• GPT-5.4: Largest expert-recognized gain (+1.25).
• GLM-5.1: Moderate positive effect (+0.50).
• DeepSeek-V4 Pro, MiniMax-M2.7: Mild benefits (+0.25).
• Kimi K2.6: Strong non-expert improvement (+1.68), expert-neutral.
• Claude Sonnet 4.6: Mild negative effect or neutrality.

Crucially, these contrasts are descriptive, not inferential, due to small cell sizes.

Figure 4: Model-specific differences between skill-augmented and native-AI outputs for expert and non-expert scores, illustrating heterogeneity in skill effects across architectures.

The reliability of these model-specific findings is diminished by low expert agreement and inadequate statistical power for model-wise inference, but they strongly suggest that the efficacy of skill augmentation is context- and backbone-dependent.

Methodological and Practical Implications

The study highlights several critical technical implications:

• Workflow Integration: Skill augmentation appears most beneficial where native workflows are underdeveloped, particularly improving task decomposition, methodological completeness, validation logic, and stepwise integration in weaker LLMs.
• Skill Routing and Orchestration: The impact of skills is mediated by the agent’s ability to select and sequence skills adaptively given task requirements and data context, not by skill exposure alone.
• Human Evaluation Noise: Significant inter-rater variation among domain experts constrains the interpretability of marginal skill-induced differences. This observation mandates enhanced evaluation infrastructure, including rater calibration, consensus-building methods, and dimension-specific scoring metrics (e.g., biological validity versus structural completeness).
• Biological Validity: The study does not address the correctness or translational value of generated biomarkers; quality scores may diverge from true scientific utility without independent biological adjudication.

Limitations and Future Directions

Several limitations are noted:

• Sample Size and Power: The dataset (21 outputs) and model coverage are insufficient for confirmatory statistical inference or generalization.
• Platform-Dependence: OpenClaw was the exclusive agent environment, conflating skill effect with agent-specific behaviors.
• Rater Calibration: Negative or near-zero ICCs reflect poor rater agreement; future studies must include more robust rater design.
• Task Generalizability: Findings are constrained to a single, complex biomedical benchmark; extrapolation to other verticals or agentic domains is unwarranted without further evidence.

Future research should emphasize reliable, large-scale, multi-expert evaluation, cross-agent and cross-platform replication, inclusion of placebo or irrelevant-skill conditions, and direct biological validation of research outputs. Skill orchestrators should actively reason about analysis feasibility, procedural dependencies, and context-aware skill invocation.

Conclusion

The results provide qualified evidence that medical-research skill augmentation via agentic LLMs can directionally improve the perceived quality of complex biomedical research-analysis outputs, especially in scenarios where native LLM planning is suboptimal. However, the magnitude of the effect is limited by human evaluation noise and model-specific variability. The study illuminates key methodological hurdles—especially in human evaluation and real-world benchmarking—that must be addressed for robust, generalizable deployment of skill-augmented AI agents in biomedical research automation.