Instruction Agent Systems

• Instruction Agent is an AI system where natural language instructions drive planning, action, and constraint enforcement across diverse applications.
• Architectural patterns decompose functions into planning, execution, verification, and memory modules to tackle long-horizon, dynamic tasks.
• Key challenges include grounding instructions in changing contexts, ensuring security, and optimizing learning from real-time feedback.

Instruction agents are AI systems in which natural-language instructions serve as the operative interface for action, planning, response generation, or constraint enforcement. In recent research, the term spans conversational tutors, embodied navigation agents, GUI automation systems, recommender intermediaries, code-generation pipelines, scheduling controllers, and security frameworks that interpret or enforce agent instructions. Across these settings, the central technical problem is not merely language understanding, but the conversion of instruction into grounded behavior under partial observability, long horizons, changing context, user feedback, and safety constraints (Suhr et al., 2022, Li et al., 8 Sep 2025, Han et al., 21 Aug 2025).

1. Conceptual scope and formal problem settings

The literature does not treat instruction agents as a single task family. Instead, it instantiates instruction following under several formal regimes. In collaborative 3D environments, an instruction-following agent is modeled as a policy $\pi(\overline{x}, o; \theta)$ that maps natural-language instruction $\overline{x}$ and observation $o$ to actions such as moving, turning, and STOP, and is updated from realtime binary user feedback via a contextual bandit objective (Suhr et al., 2022). In multi-agent reinforcement learning, instruction following is formulated in an instruction-augmented Dec-POMDP, where external instructions can interrupt macro-actions and alter objectives mid-trajectory; this motivates value-corrected Bellman targets rather than naïve reward conditioning (Lin et al., 12 May 2026). In GUI automation, tasks are modeled as a POMDP $M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$, with screenshots as observations and UI operations as actions, and the initial observation is augmented with instructions extracted from expert demonstrations (Li et al., 8 Sep 2025).

Instruction optimization itself is also treated as a sequential decision problem. APOLO formulates instruction refinement for linguistic emotion diagnosis as a POMDP $$(\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{O}, \mathcal{R})$$, where states correspond to latent diagnostic cues, actions correspond to prompt interventions, and rewards are gains in diagnostic metrics such as Micro-F1 and Jaccard similarity (Zhang et al., 20 Jan 2026). This broadens the notion of an instruction agent: some systems act in an external environment, whereas others act on the instruction interface itself.

A further expansion appears in software engineering. ContextCov treats agent instruction files such as AGENTS.md as passive natural-language artifacts that can be parsed into executable constraints, moving the locus of “instruction” from user commands to developer-authored governance over an autonomous coding agent (Sharma, 28 Feb 2026). A plausible implication is that the field now includes both instruction-consuming agents and instruction-constrained agents.

2. Architectural patterns

Despite domain diversity, recurrent architectural motifs are visible. One motif is explicit modular decomposition into planning, execution, verification, and recovery. RoboGPT combines an LLM-based planner with a low-computational Re-Plan module and a RoboSkill module for navigation and manipulation, using a new robotic dataset of 67k daily instruction tasks to improve plan feasibility and generalization (Chen et al., 2023). Instruction Agent for GUI automation separates an Instructor from an Actor; the Actor further contains a UI Grounder, Executor, Verifier, and Backtracker, and follows demonstration-derived instructions step by step to reduce execution errors on long-horizon tasks (Li et al., 8 Sep 2025). CoLVLM Agent organizes multimodal reasoning into a turn-wise “memory-perception-planning-execution” cycle, explicitly updating memory and allowing iterative self-correction without extensive retraining of the underlying LVLM (Han et al., 21 Aug 2025).

A second motif is multi-agent specialization. MAFIG contains a Perception Agent and an Emergency Decision Agent, using local reasoning over affected atomic functions rather than the full scheduling context (Zhao et al., 13 Apr 2026). EHR-CoAgent divides prediction and critique between a predictive agent and a critical agent, with critic-generated instructions appended to later predictor prompts (Cui et al., 2024). CoEvol uses five agents—Positive Debater, Critical Debater, Advisor, Editor, and Judge—in a debate-advise-edit-judge loop for improving instruction-finetuning responses (Li et al., 2024). Multilingual code instruction tuning likewise uses language-specific agents with memory and reflection to synthesize cross-lingual instruction-solution pairs (Yang et al., 11 Feb 2025). APOLO extends specialization further with Planner, Teacher, Critic, Student, and Target roles in a closed-loop prompt-optimization system (Zhang et al., 20 Jan 2026).

The following architectures illustrate the dominant design space.

Framework	Core modules	Function
RoboGPT	Planner, Re-Plan, RoboSkill	Embodied long-term daily-task execution (Chen et al., 2023)
Instruction Agent	Instructor, UI Grounder, Executor, Verifier, Backtracker	GUI automation from expert demonstration (Li et al., 8 Sep 2025)
CoLVLM Agent	Memory, Perception, Planning, Execution	Multi-turn visually grounded dialogue and instruction following (Han et al., 21 Aug 2025)
MAFIG	Perception Agent, Emergency Decision Agent, SFL	Formal instruction generation for scheduling emergencies (Zhao et al., 13 Apr 2026)
EHR-CoAgent	Predictive agent, Critical agent	Few-shot EHR disease prediction with critique-driven prompt refinement (Cui et al., 2024)
CoEvol	Positive Debater, Critical Debater, Advisor, Editor, Judge	Response improvement for instruction finetuning (Li et al., 2024)

These systems show that contemporary instruction agents are rarely monolithic sequence models. They more often externalize intermediate functions—planning, reflection, verification, localization, or critique—into named modules or separate agents.

3. Grounding, memory, and personalization

Instruction following depends on how instructions are grounded in state, history, and feedback. In realtime collaborative interaction, the continual-learning framework in CerealBar converts user feedback $f_j \in \{+1,-1\}$ into immediate rewards by attributing feedback to actions within a short post-action window corrected by a response delay $d = 0.2$ seconds:

$$r_i = \sum \left\{ f_j \mid w_i^a < w_j^f - d \leq w_{i+1}^a \right\}.$$

The system further uses reward propagation up to 8 steps ahead, increasing the proportion of actions with assigned reward from $63.1\%$ to $82.2\%$ at a modest $\overline{x}$0 error rate (Suhr et al., 2022). This makes instruction grounding an online credit-assignment problem rather than a static text-to-action mapping.

Several systems ground instructions in user-specific memory. InstructAgent reranks recommender outputs according to user-authored instructions and relevant knowledge, while Instruct²Agent adds a dynamic memory mechanism that updates a user-specific profile from actual feedback such as reviews (Xu et al., 20 Feb 2025). AppAgent-Pro tracks user interaction history, including summaries, preferences, and past queries, and uses that history to support proactive subtask generation and deeper multi-domain information mining (Zhao et al., 26 Aug 2025). CoLVLM Agent maintains hierarchical short-term and long-term dialogue memory for persistent objects, user preferences, and spatial relations, updating memory after each turn with instructions, actions, and internal reasoning traces (Han et al., 21 Aug 2025).

Personalization is especially explicit in tutoring. PACE simulates student learning styles with the Felder-Silverman model, infers a teaching strategy from the student persona, and then delivers instruction through multi-turn Socratic dialogue (Liu et al., 18 Feb 2025). Its response generation is formalized as

$\overline{x}$1

where $\overline{x}$2 is the task instruction and $\overline{x}$3 is the student profile (Liu et al., 18 Feb 2025). Here, instruction following is inseparable from persona modeling.

Grounding also includes error localization. I2EDL cross-references textual instructions with past observations during navigation, uses a detection threshold $\overline{x}$4 to flag likely instruction errors, and localizes probable error tokens so that the agent can ask a focused clarification question rather than requesting a full restatement (Taioli et al., 2024). In multilingual navigation, UVLN grounds instructions from languages such as Hindi and Telugu through XLM-R, cross-modal fusion, BLIP-based vision-language pretraining, and GPT-3-based explanation of the agent’s current state and action decisions (Zhou, 2023).

A common misconception is that instruction following is equivalent to direct supervised imitation. The continual-learning results in CerealBar show instead that realtime binary feedback is roughly equivalent to supervised demonstration data as a learning signal when carefully attributed and reused across rounds (Suhr et al., 2022).

4. Domain-specific realizations

In education, instruction agents appear both as tutors and as prompt-optimization systems for mental health diagnostics. PACE is a mathematics tutor that constructs personalized teaching data from GSM8K, simulates student personas, and uses Socratic questioning to provide instant feedback and encourage deep thinking (Liu et al., 18 Feb 2025). APOLO addresses linguistic emotion diagnosis for depression, anxiety, trauma-related states, and comorbidity, using multi-agent instruction refinement to improve diagnostic robustness in clinical language settings (Zhang et al., 20 Jan 2026).

In healthcare prediction, EHR-CoAgent converts structured EHR data into natural-language narratives and uses a predictive agent plus critic agent to improve few-shot disease prediction without parameter updates (Cui et al., 2024). In recommendation, iAgent introduces a user–agent–platform paradigm in which an agent acts as a protective shield, reranking platform outputs according to free-text instructions and individual feedback rather than exposing the user directly to platform objectives (Xu et al., 20 Feb 2025).

Embodied and navigation settings remain central. I2EDL defines Interactive Vision-and-Language Navigation in Continuous Environments, where the user may provide erroneous directions and the agent can query for correction during navigation (Taioli et al., 2024). UVLN extends vision-language navigation to multilingual and low-resource-language instructions, with qualitative emphasis on accessibility for users who cannot directly observe the environment (Zhou, 2023). RoboGPT addresses daily embodied tasks through long-horizon planning, semantic mapping, and replanning under nomenclature diversity (Chen et al., 2023). MAVIC studies instruction interruptions in cooperative multi-agent environments such as Box Pushing, Warehouse Tool Delivery, and Overcooked (Lin et al., 12 May 2026).

GUI automation has become a particularly active frontier. AppAgent-Pro proposes a proactive GUI agent that infers latent needs, decomposes them into subtasks across multiple applications, and iteratively refines information gathering in shallow or deep execution modes (Zhao et al., 26 Aug 2025). Instruction Agent instead uses a single expert demonstration to extract step-by-step instructions and then follows the demonstrated trajectory strictly, with verifier and backtracker modules handling interruptions such as pop-up windows (Li et al., 8 Sep 2025).

In software and code, instruction agents operate at both data-generation and compliance layers. CoEvol improves responses in instruction-finetuning datasets through multi-agent debate and editing (Li et al., 2024). The multilingual code instruction-tuning framework generates x-Instruct through collaboration among language-specific agents with generation memory (Yang et al., 11 Feb 2025). ContextCov converts textual agent instructions into AST checks, shell shims, and architectural validators to mitigate “Context Drift” in autonomous software engineering (Sharma, 28 Feb 2026).

In scheduling, MAFIG constrains emergency handling to local functional modules, using formal instruction generation to repair scheduling logic under local failures while reducing latency via span-focused distillation (Zhao et al., 13 Apr 2026).

5. Evaluation and empirical findings

Evaluation in this area is strongly task-dependent. Some systems measure instruction execution or task success directly, others evaluate ranking quality, human judgments, or benchmark-specific composite metrics such as SIN for interaction efficiency.

System	Evaluation basis	Reported result
Continual instruction following in CerealBar	Instruction execution accuracy over 11 rounds	Improved from $\overline{x}$5 to $\overline{x}$6, a $\overline{x}$7 absolute gain (Suhr et al., 2022)
PACE	Reference-based metrics and GPT-4 qualitative judging	Average performance improvements between $\overline{x}$8 and $\overline{x}$9 over strong baselines; higher personalization, engagement, and inspiration (Liu et al., 18 Feb 2025)
Instruct²Agent	Ranking metrics across four datasets	Average $o$0 improvement over strong baselines (Xu et al., 20 Feb 2025)
MAFIG	Success rate and average processing time	Port $o$1 at $o$2 s, Warehousing $o$3 at $o$4 s, Deck $o$5 at $o$6 s (Zhao et al., 13 Apr 2026)
CoLVLM Agent	Human evaluation on MMDR-Bench	Average score $o$7, above GPT-4o at $o$8 and Gemini 1.5 Pro at $o$9 (Han et al., 21 Aug 2025)
Instruction Agent	OSWorld hard-task success rate	$M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$0 on a set of tasks that top-ranked agents failed to complete (Li et al., 8 Sep 2025)
CoEvol-trained models	MT-Bench and AlpacaEval	CoEvol-LLaMA2-7B-CHATGPT reached $M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$1 on MT-Bench and $M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$2 on AlpacaEval; CoEvol-Mistral-7B-Mixtral reached $M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$3 and $M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$4 (Li et al., 2024)

The evaluation literature also introduces domain-specific metrics that reshape what counts as good instruction following. I2EDL proposes Success weighted by Interaction Number (SIN), which penalizes unnecessary interactions relative to the number of instruction errors and thereby distinguishes efficient clarification from indiscriminate querying (Taioli et al., 2024). In recommender systems, iAgent supplements ranking metrics such as HR@1, NDCG@3, and MRR with filtered Ads metrics and popularity-weighted metrics to study echo chambers and ad removal (Xu et al., 20 Feb 2025). In multimodal dialogue, CoLVLM is judged on six dimensions: visual entity tracking, dialogue consistency, reasoning depth, instruction adherence, error suppression, and response fluency (Han et al., 21 Aug 2025).

The empirical record also identifies structural failure modes. In MARL, naïve instruction conditioning can produce value cross-contamination because Bellman updates mix value estimates across instruction contexts when instructions interrupt macro-actions (Lin et al., 12 May 2026). In GUI automation, long-horizon precision compounds multiplicatively across steps, which is why Instruction Agent emphasizes strict adherence to demonstrated trajectories plus verification and recovery (Li et al., 8 Sep 2025). In navigation, early uncorrected direction errors cause sharp degradation, making localized clarification valuable (Taioli et al., 2024).

6. Reliability, security, and open problems

Reliability and security have become central because instruction agents increasingly operate without real-time human supervision. ContextCov addresses one reliability failure mode—Context Drift—by extracting constraints from AGENTS.md-style files and compiling them into executable checks across source, process, and architecture domains. On 723 open-source repositories, it synthesized 46,316 executable checks and achieved 99.997% syntax validity among Python-generated checks (Sharma, 28 Feb 2026). This reframes instruction compliance as a runtime-enforceable property rather than a prompt-level aspiration.

Security work shows that following instructions is not intrinsically safe. AgentSentry argues that static, over-privileged permissions make mobile agents vulnerable to instruction injection, and replaces app-wide permissions with dynamic, task-scoped policies expressed over agent, resource, operation, and context (Cai et al., 30 Oct 2025). In the demonstrated attack scenario, the framework allowed an agent to read a verification code for app registration while blocking an injected request to forward private emails, because no Allow rule existed for sending or forwarding under the task’s policy (Cai et al., 30 Oct 2025).

Multimodality introduces a further attack surface. SkillCamo hides malicious operational instructions in images bundled with skills while rewriting surrounding documentation so that the images appear integral to normal workflow. Existing scanners that focus on code, text, and manifests perform poorly against these attacks, whereas ExecScan uses intent extraction, behavior reconstruction, abuse assessment, and deliberative execution simulation across text, code, referenced resources, and visual content. In the reported experiments, ExecScan achieved precision $M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$5, recall $M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$6, and F1 $M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$7, substantially above other scanners; removing visual analysis caused detection to collapse from $M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$8 to $M = (\mathcal{S}, \mathcal{O}, \mathcal{A}, T, R)$9 in the ablation (Jia et al., 16 Jun 2026). This suggests that instruction agents cannot be secured by text-only screening once deployment-time reasoning is multimodal.

Open problems are explicit across the surveyed systems. The continual-learning framework in CerealBar notes unresolved questions about richer feedback than $$(\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{O}, \mathcal{R})$$0, broader user populations beyond English-majority locales, more complex RL objectives, and transfer beyond the single CerealBar environment (Suhr et al., 2022). CoLVLM reports a latency trade-off—approximately 4 seconds per turn versus approximately 1.5 seconds for GPT-4o—despite stronger reasoning and instruction adherence (Han et al., 21 Aug 2025). AppAgent-Pro frames most existing agents as purely reactive and therefore limited in effectiveness and efficiency for general-purpose information acquisition, implying continued work on proactive behavior and sufficiency estimation (Zhao et al., 26 Aug 2025).

Taken together, the research literature presents instruction agents as a systems field rather than a single modeling trick. The defining questions are how instructions are represented, grounded, optimized, remembered, verified, enforced, and defended. Current evidence shows that performance improvements increasingly come not from larger backbones alone, but from architectural decomposition, feedback attribution, user-specific memory, demonstration grounding, local decision scoping, and execution-level safety mechanisms (Suhr et al., 2022, Li et al., 8 Sep 2025, Zhao et al., 13 Apr 2026, Sharma, 28 Feb 2026, Jia et al., 16 Jun 2026).