Follow-Your-Instruction: Adaptive Conditional Systems

• Follow-Your-Instruction is a paradigm that uses natural language directives as task-defining signals to modulate model outputs without changing weights at inference time.
• It leverages phased fine-tuning, dynamic attention control, and prompt optimization to adapt behavior in domains like speech recognition, navigation, and retrieval.
• Recent studies demonstrate improved accuracy, robustness, and generalization, highlighting its impact on multimodal synthesis and user-aligned task execution.

Follow-Your-Instruction denotes a class of instruction-conditioned systems in which natural-language directives determine what a model should output, retrieve, generate, or execute, rather than merely providing auxiliary context. In the recent literature, this paradigm appears in speech recognition, LLM control, embodied navigation and simulated-world control, retrieval, and multimodal data synthesis. The common thread is that the instruction is treated as a task-defining signal that modulates a conditional mapping such as text from speech, actions from perception, or ranked documents from a corpus, often without changing model weights at inference time (Lai et al., 2023, Shah et al., 2018, Oh et al., 2024, Zhou et al., 2024, Feng et al., 7 Aug 2025).

1. Scope and formalization

Across the literature, Follow-Your-Instruction is not a single architecture but a family of task formulations in which the same system must change behavior as the instruction changes. In instruction-following speech recognition, the model conditions on audio $A$ and free-form instruction $I$ to produce text $T$, formalized as $P(T \mid I, A; \theta)$; the requested behavior may be verbatim transcription, word deletion, partial transcription, summarization, or immediate termination (Lai et al., 2023). In instruction-following navigation, the policy maps natural-language instructions together with egocentric vision and depth to action values $Q(o_t, a)$, with no explicit map or goal coordinates (Shah et al., 2018). In instruction-following retrieval, relevance becomes conditional on both query and instruction, so the retrieval problem is naturally written over triplets $(I,Q,P)$ rather than the conventional $(Q,P)$ pairs (Oh et al., 2024, Zhuang et al., 27 May 2025).

Domain	Instruction-conditioned object	Representative works
Speech	Text output from audio	(Lai et al., 2023, Lee et al., 2 Jun 2025)
LLM inference	Attention allocation or prompt choice	(Venkateswaran et al., 17 May 2025, Lin et al., 2023)
Embodied control	Actions from perception and language	(Shah et al., 2018, Zhou et al., 2024, Volovikova et al., 22 Apr 2026)
Retrieval	Relevance and ranking	(Oh et al., 2024, Weller et al., 31 Jan 2025, Zhuang et al., 27 May 2025)
World synthesis	Assets, layouts, scenes, future frames	(Feng et al., 7 Aug 2025)

A notable consequence of this formulation is that instruction following is often broader than task selection. Several systems must infer not only which task to perform, but also constraints on output language, structure, temporal scope, or omission behavior. This is explicit in speech systems that can be told to ignore audio or transcribe only part of an utterance, in retrieval benchmarks where the same query should retrieve different passages under different user-aligned instructions, and in verifiable instruction-following benchmarks where correctness depends on exact output constraints rather than topical adequacy alone (Lai et al., 2023, Oh et al., 2024, Pyatkin et al., 3 Jul 2025).

2. Training, alignment, and control mechanisms

One major line of work studies how instruction-following behavior is induced or amplified. Phased Instruction Fine-Tuning argues that instruction learning is gradual rather than one-off: instructions are scored by GPT-4 for difficulty, partitioned into ordered stages, and used for sequential uptraining from easy to hard. Across Llama-2 7B/13B/70B, Llama-3 8B/70B, and Mistral-7B on Alpaca-style data, this phased regime outperforms one-off instruction fine-tuning, while random stage splits or reversed curricula do not (Pang et al., 2024). A different result is that explicit instruction-response data is not strictly necessary: fine-tuning on “non-instructional” data built by splitting OpenWebText articles and using GPT-3.5-turbo or GPT-4-turbo to continue them produces strong instruction-following gains, including a LLaMA-3-70B-Instruct variant reported as comparable with LLaMA-3.1-70B-Instruct on Arena Hard (Xie et al., 2024).

A second line studies direct control over how models attend to instructions at inference time. SpotLight modifies transformer self-attention by identifying user-specified instruction spans $\mathcal{S}$, measuring the current attention proportion to those spans, and adding a logit-space bias whenever the current proportion falls below a target $\psi_{\text{target}}$. Because the bias is recomputed for every query token, head, layer, and decoding step, it is dynamic rather than fixed. The method improves instruction following on IFEval, ManyIFEval, MT-IFEval, ComplexBench, WildJailbreak, and CoCoNot, without retraining and with modest latency overhead (Venkateswaran et al., 17 May 2025). INSTINCT approaches the same problem from the prompt side: it treats instruction design as a black-box optimization problem, uses a neural bandit surrogate coupled to transformer hidden states, and outperforms APE and GP-based InstructZero on instruction induction tasks and zero-shot chain-of-thought prompt optimization (Lin et al., 2023).

A third line addresses instruction hierarchy and system-level control. RNR generates synthetic data with diverse roles and rules derived from existing instruction data, then trains models to follow system prompts rather than only user instructions. In reported experiments on Alpaca and Ultrachat, this raises rule-adherence pass rate by over 25% without regression on popular instruction-following benchmarks (Wang et al., 2024). Taken together, these works suggest that Follow-Your-Instruction behavior can be improved by curriculum design, non-instructional distillation, inference-time attention control, prompt optimization, and explicit training on developer-specified roles and rules.

3. Speech and speech-language systems

Instruction-following speech recognition provides one of the clearest modality-specific realizations of the paradigm. A Listen-Attend-Spell model with a Conformer-L encoder and 12-layer Transformer decoder is trained from scratch on LibriSpeech 960h using decoder-side natural-language instruction prefixes of the form $Y = I \oplus \text{[EOT]} \oplus T$. The model is trained over five skill families: speech transcription, ignore speech, word replacement, transcript manipulation, and summarization or keyword spotting. Despite having no pre-trained LM and no pre-trained acoustic model, the 224M-parameter instruction-trained LAS reaches 2.6 / 5.6 WER on LibriSpeech test-clean / test-other, compared with 3.1 / 6.0 for a vanilla LAS of the same size, and it achieves 100% skill-selection accuracy on seen instructions and roughly 80% on unseen instructions in the reported test suite (Lai et al., 2023).

That work is notable because the instruction changes the meaning of recognition itself. The same utterance can trigger full transcription, selective truncation such as “Only write the first half. Delete the rest.”, lexical substitution such as replacing “the” with “quokka”, summary-like keyword extraction, or immediate emission of [EOS] under “Ignore the audio in this clip.” The paper also frames these behaviors as a privacy and safety mechanism, since selective transcription or “turn off listening” semantics can prevent unwanted textual recording at the point of recognition rather than through later filtering (Lai et al., 2023).

A more modular speech-language design appears in the IWSLT 2025 instruction-following short-track submission. There, English speech is encoded by the SeamlessM4T-v2-large speech encoder, every three consecutive frames are averaged, and a 4-layer transformer projector maps 1024-dimensional speech features into the 4096-dimensional embedding space of Llama-3.1-8B-Instruct. In parallel, LoRA adapters with rank $I$0, $I$1, and no dropout are trained on text-only MT and QA. The projector and LoRA modules are then jointly loaded and instruction-tuned for 1K steps on mixed speech and text tasks so that a single model can perform ASR, speech translation into German, Italian, or Chinese, and spoken question answering from English speech conditioned by prompt templates such as “Question: Können Sie den Inhalt der Rede in den deutschen Text übersetzen?” (Lee et al., 2 Jun 2025).

The reported results show that the joint speech-text tuning is critical for complex instruction following from speech. The best multimodal variants achieve ASR WER around 18.6–18.7 on ACL 60-60, ST BLEU up to 30.75 for en-de, 40.48 for en-it, and 42.51 for en-zh, and spoken QA accuracies as high as 91.1% for en-en, 87.3% for en-de, 84.8% for en-it, and 78.0% for en-zh under LLM-as-judge evaluation (Lee et al., 2 Jun 2025). This suggests that speech instruction following can be realized either by end-to-end instruction-conditioned sequence modeling or by modular speech-to-LLM alignment, provided the LLM itself is adapted to interpret speech-conditioned prompts rather than only text prompts.

4. Embodied control and world interaction

In embodied settings, Follow-Your-Instruction concerns action selection under partial observability rather than text generation. FollowNet formulates natural-language navigation as a POMDP over instructions, semantic segmentation, and depth. The architecture uses a bidirectional GRU over the instruction, CNN encoders for segmentation and depth, and an attention mechanism over instruction tokens conditioned on the current perceptual state. A Deep Q-Network then predicts values over three locomotion primitives—turn left $I$2, go straight, and turn right $I$3. On novel two-step instructions, FollowNet reaches 52% fully successful episodes versus 40% for a no-attention baseline, a relative improvement of about 30%, and the attention heatmaps show shifting focus across words such as “left”, “hallway”, and “doorway” as the agent progresses (Shah et al., 2018).

MineDreamer addresses a harder open-world control problem in Minecraft by introducing Chain-of-Imagination. At each stage, an Imaginator conditioned on the current observation and instruction generates a plausible next-state image, a Prompt Generator converts current image, imagined image, and text into a latent visual prompt, and a VPT-based PolicyNet combines that prompt with the visual embedding to produce low-level keyboard-and-mouse actions. The system is trained on a Goal Drift dataset built from event-aligned Minecraft trajectories, and it is evaluated on single-step tasks such as “chop a tree” and “collect dirt” and on command-switching tasks such as “dig down” followed by horizontal diamond mining. The paper reports that MineDreamer nearly doubles the best generalist baseline and substantially outperforms both VPT and STEVE-1(text) on steady instruction following (Zhou et al., 2024).

SuperIgor shifts the focus from reactive control to planner-executor co-training. A LLM first extracts normalized subtasks and prerequisite relations, builds an ontology over those subtasks, and expands an instruction into a plan. A PPO-trained goal-conditional policy then executes one plan step at a time in a stochastic, partially observable environment, with a DONE action controlling subtask transitions. Plan success rates $I$4 are converted into preference pairs, and the planner is updated by DPO so that it prefers plans the RL agent can actually execute. On FOCUS/CrafText, SuperIgor substantially outperforms instruction-only baselines on atomic and combo tasks while also showing strong generalization to paraphrases and new object combinations (Volovikova et al., 22 Apr 2026).

Follow-Your-Instruction also appears in world-data synthesis rather than control. A multimodal LLM agent can collect assets, generate 3D layouts, refine semantics with multi-view VLM scoring, and plan temporally coherent future frames for 2D, 3D, and 4D synthetic data generation. The reported system includes an MLLM-Collector, MLLM-Generator, MLLM-Optimizer, and MLLM-Planner, and downstream fine-tuning on its synthetic data improves object removal, multi-view reconstruction, and camera-controlled video generation. For ReCamMaster, the added synthetic data improves FID from 62.48 to 60.32, FVD from 160.72 to 155.71, RotErr from 1.45 to 1.35, and TransErr from 5.22 to 4.69 (Feng et al., 7 Aug 2025). A plausible implication is that instruction following in multimodal agents is increasingly tied to explicit scene construction and verification loops, not only to direct action policies.

5. Retrieval and information access

Instruction-following retrieval departs from the standard query-only setup by making relevance conditional on user context, constraints, and preferences. INSTRUCTIR constructs a benchmark in which the same query is paired with multiple user-aligned instructions and different rewritten targets, so the retriever must distinguish not only topic but intended search context. It introduces Robustness@10, defined by grouping instances with the same query and different instructions, taking the minimum nDCG@10 within each group, and averaging across groups. Under this evaluation, retrievers fine-tuned on task-style instructions can underperform non-instruction-tuned counterparts; for example, INSTRUCTOR variants fall well below corresponding GTR models on both nDCG@10 and Robustness@10, which the paper interprets as overfitting to narrow task-description formats rather than robust user-aligned instruction following (Oh et al., 2024).

The multilingual extension of this problem is made explicit by mFollowIR, which uses edited TREC NeuCLIR narratives as instructions and evaluates retrieval in cross-lingual En-XX and multilingual XX-XX settings for Persian, Chinese, and Russian. The key metric is p-MRR, which measures whether documents that become newly non-relevant under an edited instruction are demoted in rank. The main finding is asymmetrical transfer: English-instruction-trained retrievers can show strong cross-lingual instruction following, but performance drops notably when the instructions themselves are in non-English languages, indicating that multilingual instruction-following retrieval remains underdeveloped (Weller et al., 31 Jan 2025).

InF-IR addresses the training-data side. It expands MS MARCO pairs into more than 38,000 positive $I$5 triplets and adds two structured hard negatives per positive by poisoning instructions and queries, with validation by o3-mini to preserve semantic plausibility while ensuring instructional incorrectness. The companion InF-Embed model uses instruction-aware embedding construction and multivariate contrastive learning over passages, instructions, and instruction-aware queries. Across five instruction-based retrieval benchmarks, the paper reports that InF-Embed surpasses competitive baselines by 8.1% in p-MRR (Zhuang et al., 27 May 2025). This suggests that retrieval models require not only instruction-aware evaluation but also training data in which instructions actively reshape the geometry of relevance.

6. Evaluation, failure modes, and open problems

A recurring result across the literature is that strong in-domain instruction-following scores do not imply robust generalization. “Generalizing Verifiable Instruction Following” shows that many models strongly overfit small benchmark taxonomies of output constraints, especially IFEval’s 25 templates. The paper introduces IFBench with 58 new hand-designed verifiable out-of-domain constraints and reports that even strong models struggle there. It also shows that reinforcement learning with verifiable rewards, implemented with carefully designed verification modules and GRPO-style training, substantially improves precise instruction following and generalization to unseen constraints (Pyatkin et al., 3 Jul 2025).

A second failure mode is conflict between explicit instructions and in-context patterns. “Do as I Say, Not as I Do” constructs conversations where the user instruction specifies a target behavior $I$6 while many hardcoded assistant turns demonstrate a competing pattern $I$7. Across 13 models and 16 instruction conditions, average instruction-following rates range from 1% to 99%, the transition from instruction-following to pattern-following is universal but highly model-dependent, and performance is largely uncorrelated with standard capability benchmarks in the fixed-output regime. The paper also shows that chain-of-thought reasoning improves robustness but does not eliminate susceptibility, and that models systematically underestimate their own resistance to induction pressure when asked to predict their behavior (Camassa et al., 19 May 2026). This directly challenges the assumption that instruction following is stable once a system prompt or user instruction has been given.

Several mitigation strategies recur across papers, but each comes with conditions. Dynamic attention steering improves adherence without retraining, yet requires white-box access to attention logits and careful selection of the target attention proportion $I$8 to avoid over-steering or over-refusal (Venkateswaran et al., 17 May 2025). Phased IFT improves adherence by ordering training data from easy to hard, but depends on external difficulty scoring and manual phase boundaries (Pang et al., 2024). Non-instructional fine-tuning can induce or strengthen instruction-following behavior without explicit instruction data, but precise format-level adherence remains weaker than in dedicated instruction-tuned models on benchmarks such as IFEval (Xie et al., 2024). In speech, unseen instructions with out-of-distribution vocabulary such as “annihilate” instead of “ignore” can fail even when semantically nearby prompts succeed (Lai et al., 2023). In retrieval, English-only instruction tuning transfers better to English-mediated cross-lingual settings than to truly multilingual instruction settings, implying a persistent data bottleneck (Weller et al., 31 Jan 2025).

Taken together, the literature presents Follow-Your-Instruction as a general capability of conditional adaptation rather than a narrow benchmark skill. The strongest systems treat instructions as objects that can be encoded, optimized, verified, grounded in perception, translated into plans, or emphasized at inference time. The dominant open problems are generalization beyond fixed instruction distributions, robustness under long context and conflicting exemplars, multilingual and multimodal coverage, and balancing exact adherence to developer or user constraints with overall task quality and harmlessness (Pyatkin et al., 3 Jul 2025, Camassa et al., 19 May 2026, Wang et al., 2024).