---
title: 'InstructFX2FX: Session-Aware Audio FX Control'
url: https://www.emergentmind.com/topics/instructfx2fx
type: topic
---

# InstructFX2FX: Session-Aware Audio FX Control

Searching arXiv for the target paper and closely related prior work.
First, retrieving the target paper by arXiv ID.
Using arXiv search for: 2606.22005
InstructFX2FX is a session-aware system for controlling audio effects through multi-turn natural-language interaction. Rather than treating text-to-FX as a single-shot mapping from one descriptor to one preset, it formulates the task as iterative refinement of an existing effect chain under successive instructions, such as “make it warmer” followed by “too harsh, soften it.” The central claim is that practical audio engineering is stateful: each new instruction should update the current chain and parameter state while preserving prior, already-satisfactory changes. To address that setting, InstructFX2FX combines a large language model as a high-level planner with CLAP-guided optimization over rendered audio, and exposes the refinement trajectory to the operator through checkpoint auditioning and strength selection [2606.22005].

## 1. Problem formulation and conceptual scope

InstructFX2FX formalizes multi-turn effect editing as a sequential state update. Given dry input audio $x$, current chain $C_{t-1}$, current parameters $P_{t-1}$, prior instruction history $H_{t-1} = \{I_1,\dots,I_{t-1}\}$, and a new instruction $I_t$, the system updates the session state according to

$$
(C_t, P_t) = f(x, C_{t-1}, P_{t-1}, I_t, H_{t-1}).
$$

This contrasts with the one-shot formulation

$$
P = g(x, I),
$$

in which each prompt is treated independently and prior edits are not represented as persistent state.

The distinction is substantive rather than cosmetic. In the single-shot regime, re-prompting on every turn can regenerate parameters non-deterministically, change parameters unrelated to the new instruction, and operate without access to the currently rendered sound. InstructFX2FX instead defines the task as sequential refinement relative to the current auditory result. This makes the target behavior closer to ordinary engineering practice, where a later instruction is interpreted as a correction or extension of an already existing sound design rather than as a request for wholesale preset replacement.

A common misunderstanding is to equate multi-turn editing with repeated single-turn text-to-preset inference. InstructFX2FX rejects that equivalence. Its novelty lies not simply in accepting more than one prompt, but in representing and updating a persistent chain-and-parameter state across turns.

## 2. Hybrid architecture and persistent session state

The system is organized around three components: an LLM planner, a routing module, and a CLAP-guided optimization backend. The LLM planner selects effects, determines signal-chain order, and proposes initial parameter values. The routing module reads the persistent session state and decides how the new instruction should interact with the current chain. The optimization backend then refines parameters in CLAP embedding space, starting from the planner’s initialization or from the current session state, depending on routing.

The persistent session state stores the chain $C_t$, the parameter set $P_t$, the instruction history $H_t$, and a series of optimization checkpoints consisting of parameter snapshots and rendered audio along the refinement trajectory. This state representation is what makes the system iterative. A new instruction is applied to the current chain and parameters rather than to a blank slate.

The LLM is assigned the discrete, structural decisions for which symbolic reasoning is useful: effect selection, chain ordering, and parameter initialization in a structured representation containing effect type, order index, and named parameters with values in valid ranges. No task-specific fine-tuning is reported; the model is used via prompting or in-context examples.

The backend divides effects by differentiability.

| Effect category | Effects | Optimization |
|---|---|---|
| Differentiable | EQ, reverb | Gradient descent |
| Non-differentiable | compressor, distortion, delay, bitcrush, pitch shift | Bayesian optimization |

This division is important because the contribution is not a new optimizer but a session-aware integration of symbolic planning and audio-conditioned refinement. The system uses a LAION-CLAP-style model for text and audio embeddings, while the LLM supplies the initial structured hypothesis about how the chain should look.

## 3. Session-aware routing and multi-turn behavior

The routing module mediates between prior state and new instruction. It implements three operating modes.

**Initialize-only** is used when no chain exists yet. The LLM creates a new chain and parameterization from scratch.

**Mixed reuse-and-initialize** is used when the current chain remains partly relevant but new effects should be added. Existing effects are retained and new ones proposed by the LLM are inserted.

**Reuse-and-optimize** is used for corrective or subtle refinement instructions. The chain is kept fixed and only parameters are adjusted by optimization.

This routing logic operationalizes the state update equation. An additive instruction such as “add some grit” is treated differently from a corrective instruction such as “too harsh, soften it.” The former may extend the chain; the latter is interpreted as a local edit over the current sound. The significance of this design is that prior intent persists unless the instruction implies structural change. Corrective prompts therefore become local parameter updates rather than triggers for complete regeneration.

The LLM remains the “brain” of the system only at the structural level. Once a reasonable chain and initialization exist, CLAP-guided refinement acts on the actual rendered sound. This reduces the parameter drift associated with LLM-only re-prompting, because optimization begins from the current parameter state and stays local to it.

## 4. CLAP-guided optimization objectives

CLAP provides a joint embedding space through an audio encoder $\phi_{\text{audio}}(\cdot)$ and a text encoder $\phi_{\text{text}}(\cdot)$. InstructFX2FX uses CLAP as an oracle for whether the current processed audio sounds more like the instruction or descriptor transition.

For direct text-audio alignment, the system uses the semantic loss

$$
\mathcal{L}_{\text{sem}} = 1 - \cos\!\bigl(\phi_{\text{audio}}(\mathbf{x}), \phi_{\text{text}}(I)\bigr).
$$

For sequential descriptor transitions from source descriptor $A$ to target descriptor $B$, it uses the directional loss

$$
\mathcal{L}_{\text{dir}} = 1 - \cos\!\bigl(\phi_{\text{audio}}(\mathbf{x}(t)) - \phi_{\text{audio}}(\mathbf{x}_0),\ \phi_{\text{text}}(B) - \phi_{\text{text}}(A)\bigr).
$$

This objective encourages the change in audio embedding to align with the semantic change from $A$ to $B$ rather than merely maximizing similarity to a single endpoint descriptor. For instructions that can be decomposed into attraction and repulsion terms, the system also supports

$$
\mathcal{L}_{\text{guided}} = 1 - \cos\!\bigl(\phi_{\text{audio}}(\mathbf{x}), \phi_{\text{text}}(I^+)\bigr) + \cos\!\bigl(\phi_{\text{audio}}(\mathbf{x}), \phi_{\text{text}}(I^-)\bigr).
$$

The practical meaning of these objectives is that CLAP refinement is driven by the audio that is actually rendered, not by symbolic prompt interpretation alone. For differentiable effects such as EQ and reverb, optimization proceeds through gradient descent on one of these CLAP-based losses. For non-differentiable effects such as compressor, distortion, delay, bitcrush, and pitch shift, the loss is treated as a black box and optimized with Bayesian optimization.

A key property of the method is locality. Because optimization starts from the current parameter state, the refinement process is less likely to change unrelated parameters than LLM-only regeneration. This is the mechanism by which the system attempts to preserve earlier successful edits while still responding to new language input.

## 5. Interactive demo design and user-in-the-loop control

InstructFX2FX is presented primarily as a demo system rather than as a purely offline benchmark. The interaction loop is designed so that an operator remains in the decision process after each turn. A dry recording is processed in response to an instruction, the system saves optimization checkpoints along the trajectory, and the interface exposes those checkpoints through a slider or similar control. The operator then auditions intermediate renders and selects how strongly the effect should be applied before issuing the next instruction.

This checkpoint mechanism is not merely a user-interface convenience. It is motivated by the empirical observation that optimization trajectories can overshoot perceptually satisfactory states or deteriorate later in the search. Early checkpoints may therefore be preferable to the final iterate. The slider externalizes this uncertainty and lets the operator choose the preferred tradeoff between under- and over-application.

The demo workflow reflects the three routing modes. An initial request such as “make it sound like a church” triggers chain initialization, typically centered on reverb. A later request such as “add some grit” leads to mixed reuse-and-initialize, in which a distortion stage may be appended while the reverb is retained. A corrective request such as “too harsh, soften it” invokes reuse-and-optimize, preserving the chain but refining parameters. The chosen checkpoint becomes the next session state.

The implementation is not yet real-time and is not integrated as a VST/AU plugin. Optimization incurs several seconds of latency per turn. That limitation is explicit and frames the system as a proof-of-concept for session-aware language control rather than as a finished production plug-in.

## 6. Evaluation protocol and reported results

The quantitative evaluation uses SocialFX-derived descriptor transitions, principally for EQ, because that dataset provides comparatively clean mappings between descriptors and parameter settings. Seven EQ descriptors are selected by frequency and consistency criteria: warm, bright, soft, loud, harsh, calm, and heavy. From these, the authors construct 10 directed descriptor pairs such as warm $\rightarrow$ bright, harsh $\rightarrow$ soft, heavy $\rightarrow$ calm, calm $\rightarrow$ loud, and loud $\rightarrow$ heavy [2606.22005].

For each descriptor and instrument condition, piano and violin, 50 parameter instances are sampled from SocialFX EQ parameter distributions. Ground-truth sequential transitions are then built by applying descriptor-$A$ parameters to dry audio to obtain $x_A$, and descriptor-$B$ parameters on top of $x_A$ to obtain $x^{\text{GT}}_{A\rightarrow B}$. Evaluation is based on distributions rather than on a single target rendering.

The metric is Maximum Mean Discrepancy with a Gaussian kernel, computed over 35-dimensional DSP feature vectors that include spectral descriptors such as centroid, bandwidth, and rolloff, along with MFCCs and temporal features. Lower MMD indicates that the generated distribution is closer to the ground-truth distribution. Because optimization is performed in CLAP space but evaluation is performed in DSP-feature space, improvements in MMD indicate transfer from embedding-space refinement to more conventional signal-level descriptors.

The main sequential result is that CLAP-guided refinement lowers target-directed DSP-feature MMD on 9 of 10 directed descriptor pairs relative to an LLM-only reprompting baseline. The largest improvements are reported for pairs with strong timbral contrast, including heavy $\rightarrow$ calm and harsh $\rightarrow$ soft, while calm $\rightarrow$ loud is described as the smallest or a negative improvement case. Trajectory analysis further shows that some pairs improve stably, as in loud $\rightarrow$ heavy, whereas others exhibit overshoot, as in calm $\rightarrow$ loud for violin, where MMD initially worsens for approximately 400 iterations before later dropping below the baseline.

A separate ablation on eight single descriptors, the seven above plus happy, compares LLM initialization alone with LLM initialization followed by CLAP gradient descent. Average MMD decreases from 0.4461 without CLAP to 0.3380 with CLAP refinement, approximately a 24% reduction. In CLAP space, refined solutions surpass the initialization for 7 of 8 descriptors. No formal listening tests are reported.

## 7. Limitations, relation to prior work, and implications

The paper situates InstructFX2FX against two existing tendencies in text-guided audio FX: single-turn CLAP-based optimization and single-turn LLM-based text-to-parameter mapping. Its contribution is to recast the task as stateful multi-turn refinement, to combine LLM planning with CLAP refinement, to introduce a routing layer that chooses between initialization, extension, and reuse, and to expose optimization checkpoints to the operator. The directional CLAP objective for descriptor transitions is another explicit contribution, because it models “from descriptor $A$ to descriptor $B$” as a vector in embedding space rather than as a flat prompt.

Several limitations are stated directly. Quantitative experiments focus mainly on EQ-related descriptors, so broader behavior on reverb, compression, distortion, and more interaction-heavy chains is less systematically characterized. CLAP does not perfectly capture all perceptual aspects relevant to audio production, and there is an acknowledged mismatch between CLAP objectives and DSP-feature MMD; improvements in one space do not always correlate monotonically with the other. For non-differentiable effects, Bayesian optimization can produce noisy and semantically inconsistent trajectories, and corrective prompts after distortion are described as particularly unstable. Latency remains high enough to preclude real-time DAW integration [2606.22005].

These limitations shape the paper’s proposed future directions: broader multi-effect chains, human listening studies, domain-adapted FX embedding models beyond general CLAP, more stable optimization methods for non-differentiable effects, real-time VST/AU integration, more complex multi-track scenarios, and eventually generative sequential parameter models such as flow-matching or diffusion over parameter trajectories trained on real editing sessions. This suggests that InstructFX2FX is best understood as a proof-of-concept formulation of sequential language-driven audio effect control: not a complete solution to interactive audio engineering, but a concrete and technically explicit demonstration that hybrid symbolic planning and audio-perceptual optimization can be organized around persistent session state.

Source: https://www.emergentmind.com/topics/instructfx2fx