A Flexible Multi-Agent LLM-Human Framework for Fast Human Validated Tool Building
Abstract: We introduce CollabToolBuilder, a flexible multiagent LLM framework with expert-in-the-loop (HITL) guidance that iteratively learns to create tools for a target goal, aligning with human intent and process, while minimizing time for task/domain adaptation effort and human feedback capture. The architecture generates and validates tools via four specialized agents (Coach, Coder, Critic, Capitalizer) using a reinforced dynamic prompt and systematic human feedback integration to reinforce each agent's role toward goals and constraints. This work is best viewed as a system-level integration and methodology combining multi-agent in-context learning, HITL controls, and reusable tool capitalization for complex iterative problems such as scientific document generation. We illustrate it with preliminary experiments (e.g., generating state-of-the-art research papers or patents given an abstract) and discuss its applicability to other iterative problem-solving.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper introduces CollabToolBuilder, a system where several AI “agents” work together with a human expert to quickly build useful tools—like code scripts or writing helpers—that solve complex tasks. The main test in the paper is making long, fact-checked scientific documents (such as surveys or patent summaries) from just a title and abstract. The key idea is to mix the speed and creativity of AI with the judgment and knowledge of a human to get better results, faster.
What was the goal?
The researchers wanted to answer simple questions:
- Can multiple AI helpers, guided by a human, build reliable tools more quickly than AI alone?
- Can this setup produce well-structured, fact-checked documents that match real scientific papers?
- How should humans best work with AI—when to step in, what to correct, and how to guide the process?
They also shared resources: a dataset for testing scientific document generation, the CollabToolBuilder system, and a “HumanLLM” library to plug human feedback directly into AI prompts.
How did they do it?
Think of the system like a small team with defined roles, plus a human coach who can jump in at any time. The four AI agents are:
- Coach: Plans the next step, like deciding “we need a tool to find relevant papers.”
- Coder: Writes and tests the code or tool that the Coach asked for.
- Critic: Checks if the tool worked as expected and points out what’s wrong.
- Capitalizer: Saves successful tools (and notes failed attempts) so they can be reused later.
These agents run in a loop: plan → build → evaluate → save, then repeat, learning as they go.
Human-in-the-loop (HITL) in plain terms
“HITL” means a human can guide the agents:
- Before they act: suggest changes, add missing info (like API keys), or choose a different plan.
- After they act: fix mistakes, pick the best answer from multiple options, or add explanations.
This keeps the AI aligned with the human’s goals and reality (like the actual system setup or rules).
Reinforced Dynamic Prompt (RDP)
A “prompt” is the instructions given to an AI agent. Here, prompts evolve over time like a growing notebook:
- They include the agent’s role and goal, current state, tasks, examples, and feedback.
- Every cycle adds new info (scores, evaluations, human notes), making the next answers smarter and better aligned.
How they tested it
- They built a dataset of 30 documents (10 scientific surveys from arXiv, 10 Wikipedia articles, 10 European patents).
- The system was asked to write full documents from a given title and abstract.
- They measured quality using “semantic similarity” (how close the AI’s writing is in meaning and structure to the real paper), plus checks on length and planning.
- They tried three modes:
- Fully automated (AI does it all)
- Fully HITL (human guides every step)
- Hybrid (human helps early, then AI runs)
They also used practical setup: GPT-4o for the Coach and Coder (stronger planning and coding), GPT-3.5 for the Critic and Capitalizer (cheaper tasks like checking and saving). An optimizer (Optuna) tuned settings like creativity (“temperature”) and number of auto-fixes.
What did they find?
The hybrid approach performed best. When humans guided early steps and AI continued afterward, results improved the most. In short:
- Early human guidance helped the system focus on the right tools and avoid pointless detours.
- The best runs had higher similarity to real documents, especially in structure and content.
- Human time mattered most during planning (Coach) and coding (Coder), not as much for checking and saving.
Beyond scores, three clear benefits of human input stood out:
- Environment grounding: Humans provided real-world details AI can’t guess (e.g., which databases are allowed, where credentials live). This quickly fixed code that couldn’t run.
- Intent sharpening: Humans turned vague goals into precise instructions, improving tool usefulness.
- Fine-grained evaluation: Humans caught subtle issues the AI Critic missed (like content that sounds right but is misleading), helping the system learn from better feedback.
They also found practical tips:
- Seeding the process with a few working code examples boosted early progress.
- Limiting auto-fix attempts saved time without hurting quality.
- If early human feedback is wrong, it can “snowball,” so there should be ways to undo bad memories.
Why is this important?
- It shows a practical way to blend human judgment with AI speed to build reliable tools.
- It helps create long, structured, and fact-checked documents—something single-pass AI often struggles with.
- It can adapt to different fields. The authors tested it not only on scientific papers but also on fixing software issues and helping with environmental tasks (like tracking pollution and planning interventions).
- It reduces “AI hallucinations” by grounding the process in real data, rules, and human oversight.
- It’s available for others to use and build on (they open-sourced the system and dataset).
Key terms explained
- LLM: An AI that understands and generates text (like ChatGPT).
- Agent: A role-specific AI helper (planner, coder, critic).
- Human-in-the-loop (HITL): A human guiding the AI during the process.
- Prompt: The instructions given to the AI.
- Reinforced Dynamic Prompt (RDP): A prompt that grows with each cycle, adding scores, feedback, and examples so the AI keeps learning.
- Semantic similarity: A score showing how similar two texts are in meaning and structure, not just exact words.
- Tool library: A collection of reusable code pieces the system has learned to build.
- Optimizer (Optuna): A helper that tries different settings to find what works best.
Final takeaways
CollabToolBuilder is like a smart workshop where AI and humans co-create tools. Giving humans a strong role early on helps the system learn faster and produce better results—even for complex jobs like writing scientific surveys. Because it saves and reuses tools, it gets more efficient over time. This approach could help researchers, engineers, and organizations build trustworthy AI-assisted workflows in many areas, from writing to software to environmental planning.
Collections
Sign up for free to add this paper to one or more collections.