InstructFX2FX: A Multi-turn Text-to-Preset Demo for Iterative Audio Effect Refinement

Abstract: We present InstructFX2FX, an interactive demonstration of multi-turn, text-guided audio effect refinement. Existing text-to-preset systems are largely single-shot, mapping one textual descriptor to one preset. Real audio engineering is instead sequential: engineers refine an existing effect chain through successive instructions such as "make it warmer" or "too harsh, soften it." This poses a stateful problem that single-shot systems do not address - given the current effect parameters and a new instruction, update the sound while preserving what earlier instructions already achieved. InstructFX2FX tackles this with a hybrid architecture that divides labor between a LLM and CLAP-guided optimization. The LLM acts as a high-level planner that selects effects, orders the signal chain, and proposes the initial parameter state, leveraging recent evidence that LLMs can outperform CLAP-based optimization for single-turn text-to-parameter mapping; CLAP-guided optimization then refines the existing parameter state from this initialization, avoiding the parameter drift caused by re-prompting the LLM at each turn. At the demo, attendees drive a dry recording through successive natural-language instructions while staying in the loop: after each turn they audition the saved optimization checkpoints to choose how strongly an effect is applied, then issue the next instruction based on what still differs from the sound they intend. On SocialFX-derived descriptor transitions, CLAP-guided refinement lowers target-directed DSP-feature MMD on 9 of 10 directed descriptor pairs relative to an LLM-only reprompting baseline, with most refinement trajectories trending toward the intended target.

• The paper presents a hybrid method that combines LLM planning and CLAP optimization to enable session-aware, multi-turn audio effect refinement.
• It introduces explicit routing modes that balance initialization and in-place refinement, achieving up to a 24% reduction in DSP-feature MMD.
• The study highlights the complementarity of symbolic and perceptual approaches while revealing challenges in handling non-differentiable effects and real-time latency.

InstructFX2FX: Session-Aware Multi-Turn Audio Effect Refinement via Hybrid LLM and CLAP Optimization

Motivation and Problem Statement

Traditional text-to-preset systems for audio effect manipulation are stateless and operate in a single-shot manner, mapping a solitary textual prompt to an audio effect preset. This ignores the realistic, iterative workflow of audio engineers, who sequentially refine audio chains with consecutive instructions—each contextualized by prior manipulations. InstructFX2FX formalizes this problem as sequential FX refinement: maintaining persistent state across turns, updating effect chains and parameters in response to each instruction, and preserving previously achieved audio characteristics (Figure 1).

Figure 1: InstructFX2FX harness, illustrating the division of roles between LLM planning, FX chain routing, and CLAP-based optimization in an iterative session.

System Architecture

The system implements a hybrid approach, dividing responsibilities between a LLM and a CLAP-guided optimization backend. The LLM acts as the symbolic planner, determining the FX chain, ordering effects, and generating initial parameters. Distinct from prior works that either strictly use LLMs for zero-shot parameter generation (Doh et al., 27 May 2025) or CLAP-based optimization for single-turn text-prompted transformations (Chu et al., 2024, Ki et al., 18 Nov 2025), InstructFX2FX integrates both in a session-aware, multi-turn loop.

For every instruction, the routing module selects among three modes: initialize-only, mixed reuse-and-initialize, or reuse-and-optimize, contingent on session state (Figure 2). This routing enables both additive and corrective instructions, extending or locally refining chains as appropriate. The optimization backend then refines parameters initialized by the LLM, using CLAP objectives (semantic alignment, directional transition, and guided constraints) via gradient descent or Bayesian optimization, depending on effect differentiability.

Figure 2: Three routing modes: initialization, mixed reuse/initialize, and in-place refinement, supporting both additive and corrective manipulations.

Optimization and Multi-Turn Interaction

The objective functions operate in CLAP embedding space, ensuring that audio effect parameter refinements are guided by the semantic transition encoded in textual instructions. Importantly, with each turn, the demo interface records intermediate optimization checkpoints, allowing users to audition varying strengths of applied effects on a slider—mitigating both overshoot and drift in optimization trajectories.

Quantitative Evaluation

Evaluation is based on the SocialFX dataset, targeting sequential EQ descriptor transitions across piano and violin conditions. The metric is Maximum Mean Discrepancy (MMD) on DSP feature vectors, providing a quantitative, feature-based comparison to sequential ground truth audio.

Results from Experiment 1 show that CLAP-guided refinement outperforms LLM-only reprompting on 9 of 10 descriptor pairs, especially where timbral contrast is pronounced (Figure 3a). A further ablation (Experiment 3) demonstrates that CLAP optimization consistently lowers DSP-feature MMD compared to LLM-initialized baselines, with an aggregate reduction of approximately 24%, indicating the refinement does not merely exploit a better starting point (Figure 3b).

Figure 3: (a) Sequential MMD improvement across descriptor pairs; (b) CLAP refinement outperforms LLM initialization on single descriptors.

Optimization trajectories (Experiment 2) reveal both monotonic improvements and typical failure modes such as early drift or overshoot, exposing a mismatch between the CLAP objective and DSP-feature metrics (Figure 4). The system interface mitigates such behavior by exposing intermediate results to users.

Figure 4: MMD convergence trajectories for prompt pairs, illustrating both successful and noisy optimization behaviors warranting user intervention.

Limitations

Several challenges persist: the system is primarily evaluated on EQ descriptors due to dataset constraints; CLAP embeddings incompletely capture perceptual timbre semantics (Deng et al., 16 Oct 2025); optimization for non-differentiable effects is noisy and unstable under corrective prompts; and interaction latency prevents real-time DAW integration.

Implications and Future Directions

Practically, InstructFX2FX advances interactive audio effect editing, approximating real-world engineer workflows within a session-aware, multi-turn architecture. Its hybrid division of labor between LLM and CLAP supports precise initialization and perceptual refinement, highlighting the complementarity of symbolic and embedding-based approaches. Theoretically, the findings expose limitations in current audio-language embedding spaces, motivating further research in domain-adapted audio-perceptual models and end-to-end generative parameter trajectories. Real-time deployment and robust handling of complex effect chains are crucial for future development.

Long-term, sequential FX refinement algorithms could underpin LLM-driven, conversational DAW interfaces, style transfer systems, and fully automated, context-aware post-production tools, leveraging generative flow-matching models for end-to-end parameter evolution.

Conclusion

InstructFX2FX introduces a hybrid architecture for iterative, text-guided audio effect refinement, aligning with practical session workflows and outperforming single-shot LLM-only systems across sequential evaluation metrics. Persistent session state, explicit routing, and CLAP-based optimization collectively enable controlled, perceptual trajectory in multi-turn editing, while exposing critical gaps in embedding representations and optimization robustness. The framework exemplifies synergy between language modeling and audio embeddings and sets the stage for deeper, more interactive machine-driven music production pipelines.