METIS: Mentoring Engine for Thoughtful Inquiry & Solutions

Abstract: Many students lack access to expert research mentorship. We ask whether an AI mentor can move undergraduates from an idea to a paper. We build METIS, a tool-augmented, stage-aware assistant with literature search, curated guidelines, methodology checks, and memory. We evaluate METIS against GPT-5 and Claude Sonnet 4.5 across six writing stages using LLM-as-a-judge pairwise preferences, student-persona rubrics, short multi-turn tutoring, and evidence/compliance checks. On 90 single-turn prompts, LLM judges preferred METIS to Claude Sonnet 4.5 in 71% and to GPT-5 in 54%. Student scores (clarity/actionability/constraint-fit; 90 prompts x 3 judges) are higher across stages. In multi-turn sessions (five scenarios/agent), METIS yields slightly higher final quality than GPT-5. Gains concentrate in document-grounded stages (D-F), consistent with stage-aware routing and groundings failure modes include premature tool routing, shallow grounding, and occasional stage misclassification.

• The paper presents a stage-aware mentorship agent that tailors guidance across six research writing stages.
• It leverages dynamic tool routing and literature retrieval to provide actionable, document-grounded advice with superior evaluation metrics.
• Evaluation shows robust improvements over baseline models in clarity, guideline compliance, and mentorship quality.

METIS: Tool-Augmented, Stage-Aware Research Mentorship Agent

System Architecture and Workflow

METIS is an interactive AI system engineered to mentor undergraduate and early-career researchers through the end-to-end research writing pipeline, segmented into six explicit stages: (A) Pre idea, (B) Idea, (C) Research plan, (D) First draft, (E) Second draft, and (F) Final. Distinct from autonomous manuscript generation agents, METIS prioritizes interactive, tool-integrated guidance grounded in retrieved literature, curated research guidelines, and context-maintaining session memory.

The architecture features a stage detector and tool router that conditionally select among the available tools—curated guideline retrieval, arXiv/OpenReview-driven literature search, methodology checks, and attachment-based document search—tailoring their invocation to the detected writing stage. This modular tool orchestration allows domain-specific interventions, greater transparency, and alignment with the learner's context across conversational turns. Every METIS reply is structured to surface two self-explanations (Intuition, Why this is principled), provides actionable next steps, and maintains explicit session memory for longitudinal progress tracking.

Figure 1: METIS system architecture showing stage detection and tool routing mechanisms that integrate literature, guidelines, methodology validation, and session memory to synthesize principled, context-aware responses.

Evaluation Protocols and Metrics

The paper introduces a comprehensive, stage-aware evaluation suite comprising 90 single-turn prompts (balanced across stages), 5 multi-turn mentorship scenarios, and diverse metric rubrics for both expert and student perspectives. Evaluation harnesses automated LLM-as-a-judge methodologies (three advanced LLMs: Gemini 2.5 Pro, DeepSeek v3.2-exp, Grok-4-fast), measuring both pairwise win rates and scalar rubric trends (clarity, actionability, constraint-fit, confidence-gain).

Pairwise preference data reveals METIS is selected over Claude Sonnet 4.5 in 71% of cases and over GPT-5 in 54%, with more pronounced advantages in document-grounded stages (D–F). Student-focused rubrics confirm sustained improvements in guidance clarity, actionability, and adherence to learner constraints, weighted so actionable next steps dominate the overall score computation.

Figure 2: LLM-judge pairwise preferences across writing stages; METIS achieves consistent preference over Claude Sonnet 4.5 and GPT-5, especially in stages where document grounding is utilized.

Figure 3: Student-judge rubric trends for METIS versus baseline systems showing superior clarity, actionable guidance, and constraint fit, especially at later research stages.

Stage-Aware Benefits and Failure Modes

METIS's architecture leads to clear stage-wise differentials:

• Early Stages (A–B): METIS provides modest improvements over baselines; all systems exhibit similar performance due to the generic nature of ideation and orientation prompts. Gains are limited as tool invocation (e.g., guidelines retrieval) is more frequent for ambiguous cases, concentrating more challenging prompts in these buckets.
• Document Grounded Stages (D–F): Dramatic improvements in both pairwise preference and rubric scores, driven by METIS’s protocol for attachment-grounded advice, rigorous methodology checks, and compliance with venue guidelines.

Evidence integrity and citation validity are high across METIS and GPT-5, with METIS exhibiting fewer grounding errors and supporting claims more faithfully—metrics including RAG fidelity and stage awareness are near optimal. Failure modes identified include premature tool routing, shallow attachment grounding, and rare stage misclassification. These failures are amenable to mitigation via improved routing heuristics or potential future transitions to learned router models.

Multi-Turn Tutoring and Scenario Outcomes

In multi-turn mentorship (five scenarios/agent; topics encompass CivicTech, healthcare AI, privacy, low-compute, etc.), METIS leads in final student quality scores, trading a modest increase in turns for higher outcome quality. The mean turns-to-success ratio highlights METIS’s thoroughness in plan elaboration and constraint fit. Scenario-level granularity indicates consistent quality gains across diverse learner types and task constraints.

Figure 4: Multi-turn mentorship quality showing higher final scores and success rates for METIS across representative scenarios, despite marginally increased interaction length.

Figure 5: Per-scenario multi-turn outcomes illustrate robust guidance quality and minor trade-offs in efficiency for METIS, confirming consistency across varied research mentorship contexts.

Human Feedback and Practical Utility

Survey data from 50 real users corroborate the system-level metrics: high ratings for ease of use (mean 4.14/5), helpfulness (4.22/5), goal understanding (4.08/5), and overall experience (4.30/5). Notably, 90% express intent to reuse METIS for ongoing research navigation, suggesting practical uptake beyond LLM-based benchmarks. Median session durations (24 hours) underscore sustained engagement, indicating that METIS scales to real multi-week mentorship requirements.

Limitations and Measurement Caveats

Evaluations using LLM-as-a-judge are known to be sensitive to position bias, verbosity, and model-family self-preference [schroeder2024reliability], and these concerns are explicitly acknowledged. The study restricts itself to text-based scholarly stages; omits experimental design, hardware, and multimodal workflows; and benchmarks only against GPT-5 and Claude Sonnet 4.5. Rubric-driven metrics are proxies for longitudinal mentorship efficacy, thus future work should seek educational impact validation over extended periods.

Implications and Future Directions

METIS demonstrates that systematic tool-integrated, stage-aware research mentorship outperforms current chat-based LLM advice in guiding students through paper development. Gains concentrate in contexts where external document retrieval and compliance heuristics are enforced through dynamic routing. This aligns with a broader migration from monolithic LLM chatbots to explicitly modular, inspectable mentoring agents with transparent routing and reasoning blocks [karpas2022mrkl, schick2023toolformer].

Immediate extensions include moving from heuristic to learned tool routers using trace logs, as well as conducting module ablations to establish causality of quality improvements. As agentic architectures and research guidelines expand, further coupling mentorship agents to knowledge discovery pipelines (e.g., Denario [villaescusa-navarro2025denario], Robin [ghareeb2025robin]) may unlock scalable, automated formative feedback for large academic cohorts. METIS’s design principles—explicit rationale blocks, progressive stage nudging, high evidential standards—are likely to inform best practices for interactive AI research pedagogy and automated tutoring.

Conclusion

METIS operationalizes interactive, stage-aware, tool-augmented AI mentorship. Empirical evaluations substantiate measurable improvements in advice quality, actionability, and constraint fit over strong LLM chat baselines. The architectural focus on stage-adaptive routing, explicit rationale surfacing, and attachment/document grounding yields robust practical and theoretical advantages. Future work should extend to curriculum-wide longitudinal learning impact, transition tool selection to trainable models, and integrate with autonomous scientific discovery pipelines.

(2601.13075)