LLM2Fx-Tools: Audio Effects Chain Framework

• LLM2Fx-Tools is a multimodal framework that converts audio and language inputs into an executable audio effects chain through explicit chain-of-thought planning.
• It integrates dry and processed audio analysis with tool calling to determine effect selection, order, and precise parameter settings for music post-production.
• The framework leverages the LP-Fx dataset and multi-stage training with LoRA fine-tuning to achieve higher effect accuracy and improved parameter estimation compared to baseline methods.

LLM2Fx-Tools is a multimodal tool-calling framework for music post-production that generates an executable audio effects chain—an ordered sequence of audio effects with concrete parameter settings—from audio and language inputs. It treats audio effects modules as callable tools and uses a LLM to infer effect types, determine their order, estimate parameters, and serialize the result as executable tool calls, with chain-of-thought planning as an explicit intermediate representation. The framework is positioned as the first work applying LLM-based tool calling to audio effects modules for music post-production, and it is accompanied by LP-Fx, a synthetic instruction-following dataset with structured chain-of-thought annotations and tool calls (Doh et al., 1 Dec 2025).

1. Definition and conceptual position

LLM2Fx-Tools addresses reverse engineering and style transfer of music production effects. In conventional post-production, engineers typically apply an ordered chain of processors such as EQ, compression, distortion, reverb, or delay, and the order is consequential because audio effects are not commutative. The framework formalizes this operational reality by modeling an effects chain as

$$\mathcal{C} = [(\text{tool}_n,\text{params}_n)]_{n=1}^N$$

and an execution environment $\mathcal{E}$ such that applying the chain to dry audio $x_{\text{dry}}$ produces processed audio $x_{\text{ref}}$:

$$x_{\text{ref}} = \mathcal{E}(\mathcal{C}, x_{\text{dry}}).$$

It also gives an equivalent compositional view in terms of successive effect functions $g_n$ with parameters $\phi_n$ (Doh et al., 1 Dec 2025).

The framework differs from fixed-topology regression systems and differentiable signal-processing approaches in three stated ways. First, its inputs are inherently multimodal: a system may receive a dry/reference audio pair, only a reference audio, and/or a natural-language instruction such as “use a stereo widener, distortion, and compressor.” Second, its output is symbolic and executable rather than latent: a serialized sequence of tool calls with numeric arguments. Third, it makes reasoning explicit by generating chain-of-thought before tool calls, thereby treating effect selection, ordering, and parameter estimation as a planning problem rather than only as direct regression (Doh et al., 1 Dec 2025).

LLM2Fx-Tools is closely related to the earlier LLM2Fx framework, which used LLMs as a text-driven control layer for individual audio effects such as EQ and reverb without task-specific training or fine-tuning. LLM2Fx formulates a Text2Fx mapping from language to DSP settings, whereas LLM2Fx-Tools extends the problem from single-effect parameter prediction to multimodal generation of an executable multi-effect chain (Doh et al., 27 May 2025).

2. Problem formulation and task settings

The principal inverse problem is written as

$$\mathcal{C} = f_{\theta}(x_{\text{instruction}}, x_{\text{dry}}, x_{\text{ref}}; \mathcal{T}),$$

and in the appendix as

$$\hat{\mathcal{C}} = f_{\theta}\Big(x_{\text{instruction}}, x_{\text{dry}}, x_{\text{ref}}; \mathcal{T}\Big),$$

where $\mathcal{T}$ is the available tool set. The model is trained to generate three outputs autoregressively: chain-of-thought $\mathcal{E}$0, tool-calling sequence $\mathcal{E}$1, and natural-language response $\mathcal{E}$2 (Doh et al., 1 Dec 2025).

Two task settings are defined. In reverse engineering, the model infers the effects chain from a dry/processed pair $\mathcal{E}$3. In blind estimation / style transfer, the model receives processed reference audio alone, constructs a pseudo-dry version $\mathcal{E}$4 via Fx-Removal and Fx-Normalization, estimates the chain, and then applies that chain to different source content (Doh et al., 1 Dec 2025).

The paper assumes, for simplicity, that $\mathcal{E}$5 when $\mathcal{E}$6. The target sequence used in supervised training is

$\mathcal{E}$7

conditioned on

$\mathcal{E}$8

Tool calls are serialized as repeated blocks. A representative example in the appendix is: $x_{\text{ref}}$5 This representation makes the output directly executable in the tool environment rather than merely descriptive (Doh et al., 1 Dec 2025).

A plausible implication is that the framework places music post-production inside the broader class of LLM-with-tools systems in which the model plans over external functions rather than only emitting free-form text. The general survey literature describes such systems as iterative mappings from user instruction to intent, plan, execution, feedback, and plan adjustment; LLM2Fx-Tools fits that description at the level of effect-chain reasoning and tool execution, although the paper itself does not formalize the process in that exact vocabulary (Shen, 2024).

3. System architecture and generation pipeline

The architecture comprises four major components: audio preprocessing, audio encoder, audio-language adapter, and LLM backbone. Audio preprocessing applies Fx-Removal and Fx-Normalization to obtain pseudo-dry audio when true dry audio is unavailable. Audio is then encoded by Fx-Encoder++, an effects-specific encoder trained with contrastive learning; the system removes its pooling and MLP projection heads and uses patch embeddings

$\mathcal{E}$9

as the audio representation (Doh et al., 1 Dec 2025).

The encoder output is mapped into the LLM token space by an adapter:

$x_{\text{dry}}$0

with

$x_{\text{dry}}$1

Rather than a single linear projection, the adapter is BLIP-2-style and uses a linear projection, 32 learnable query embeddings, and cross-attention to aggregate audio information into pseudo-token embeddings consumable by the LLM (Doh et al., 1 Dec 2025).

The LLM backbone is Qwen3-4B, selected because the paper describes it as having native capabilities for structured tool calling and chain-of-thought reasoning. Fine-tuning uses LoRA, rank 128 and alpha 256. The multimodal prefix supplied to the LLM consists of instruction token embeddings, separator tokens, dry audio embeddings, and reference audio embeddings. The separator tokens are implemented as natural-language tokens such as “dry audio” and “reference audio,” which the paper states encourages token overlap between textual context and audio segments (Doh et al., 1 Dec 2025).

Chain-of-thought is explicit and supervised rather than latent. The paper describes it as covering four sequential components: user input analysis, effect selection, processing order determination, and parameter planning. In the appendix prompt, this is further constrained into two steps: first identify the category and order of effects in the chain without exact values; then create a parameter prediction plan describing general direction and approach for each effect’s parameters. Because generation is autoregressive, the tool-call tokens are produced conditioned on the preceding chain-of-thought tokens, so planning becomes an explicit prefix for executable action generation (Doh et al., 1 Dec 2025).

This design is consistent with the broader tool-use literature’s emphasis on separating reasoning from execution. The survey literature describes a canonical pattern in which tool spans are represented explicitly and executed by an external runtime; LLM2Fx-Tools follows that pattern by generating structured tool calls over a fixed effect environment rather than embedding audio processing inside the LLM itself (Shen, 2024).

4. Tool environment and LP-Fx dataset

The tool environment contains 9 modules and 26 parameters total. The modules are: three-band equalizer, compressor, stereo widener, gain, panner, distortion, reverb, delay, and limiter. Six modules come from Pedalboard—compressor, distortion, reverb, delay, limiter, gain—and three are custom: three-band equalizer, stereo widener, and panner (Doh et al., 1 Dec 2025).

The paper reports exact sampled parameter spaces for these modules. For example, compressor parameters include threshold db, ratio, attack ms, and release ms, each with coarse and fine sampling regimes; reverb includes room size, damping, width, and mix ratio; delay includes delay seconds, feedback, and mix ratio; the equalizer includes low, mid, and high bands with gain, cutoff frequency, and Q factor. Parameters are randomly sampled within predefined min/max ranges and then quantized to discrete steps to reflect real-world knob granularity. The system uses both a coarse regime for broad coverage and a fine regime for more realistic production settings (Doh et al., 1 Dec 2025).

LP-Fx is synthesized from MedleyDB. Starting with 196 multitrack recordings, the authors filter out tracks with bleed and retain 2,119 raw audio files from 116 multitracks, spanning 9 genres and 80 unique instruments. These raw tracks are treated as dry audio. The dataset is stratified by chain length 1–9 and includes 11,100 training samples and 100 test samples for each chain length, for a total of 100,800 conversations. The statistics table reports 99,900 train dialogues, 900 test dialogues, 2,019 train tracks, 100 test tracks, 80 train instruments, 33 test instruments, average instruction length 28.8, average response length 178.6, average CoT length 251.9, and tool count min/max 1–9 (Doh et al., 1 Dec 2025).

LP-Fx is constructed in four stages. First, the system creates processed references by sampling an effects chain and applying it to dry audio after Fx-Removal and Fx-Normalization. Second, it synthesizes instruction-following conversations from the chain using Gemini-2.5-Flash-lite. Third, it generates chain-of-thought using Gemini-2.5-Flash. Fourth, it filters outputs with Gemini 2.5 Pro as judge, scoring tool alignment and CoT quality on a 4-point Likert scale and regenerating any sample scoring $x_{\text{dry}}$2 on either dimension (Doh et al., 1 Dec 2025).

A plausible implication is that LP-Fx operationalizes a domain-specific analogue of synthetic tool-use corpora used elsewhere in LLM research: the underlying action space is generated programmatically from executable tools, while conversational supervision and reasoning traces are added by stronger LLMs. The paper presents this as grounded conversational data rather than human-authored studio transcripts (Doh et al., 1 Dec 2025).

5. Training objectives and optimization

Training is based on next-token autoregression over a multimodal input sequence:

$x_{\text{dry}}$3

Cross-entropy is applied only to target tokens:

$x_{\text{dry}}$4

To improve numeric prediction, the paper adds a Number Token Loss based on Wasserstein-1 distance:

$x_{\text{dry}}$5

The combined loss is

$x_{\text{dry}}$6

The paper does not specify the value of $x_{\text{dry}}$7 (Doh et al., 1 Dec 2025).

Training proceeds in two stages. Stage 1 is modality-alignment pretraining: only the adapter is trained, the LLM is frozen, inputs are audio, and targets are tool-calling outputs only. Stage 2 initializes the adapter from Stage 1 and then fine-tunes adapter plus LLM via LoRA on full conversational data including instruction, CoT, tool calls, and response. To address the mismatch between dry/reference training pairs and realistic inference where dry audio may be absent, the system uses pseudo-dry audio and dry audio masking during training with probability $x_{\text{dry}}$8, although the paper does not specify its value (Doh et al., 1 Dec 2025).

Optimization details are reported for both stages. For Qwen3-4B, Stage 1 uses learning rate $x_{\text{dry}}$9, batch size 32, and 100K steps. Stage 2 uses learning rate $x_{\text{ref}}$0, batch size 16, and 400K steps. The paper does not specify sequence length, optimizer scheduler, or hardware beyond these settings (Doh et al., 1 Dec 2025).

The use of a specialized numeric loss is notable in the context of tool-calling systems because parameter estimation is encoded as number-token generation inside tool-call arguments. The ablation study shows that removing Number Token Loss degrades parameter MAE from 0.23 to 0.32 and worsens downstream reconstruction metrics, indicating that ordinary token cross-entropy is insufficient for precise DSP control (Doh et al., 1 Dec 2025).

6. Empirical results, interpretation, and limitations

The main reverse-engineering results report the following for LLM2Fx-Tools: effect classification accuracy 80%, order correlation 0.56, parameter MAE 0.23, left/right MRS 3.13, mid/side MRS 3.27, AF distance 8.29, AFx-Rep similarity 0.68, and Fx-Encoder similarity 0.67. Among comparison systems, Regression reaches 55% effect accuracy and -0.03 order correlation with 0.20 MAE, MultiTask reaches 61% effect accuracy and 0.00 order correlation with 0.23 MAE, Gemini 2.5 Flash reaches 78% effect accuracy, 0.54 order correlation, and 0.32 MAE, and DeepAFx-ST achieves the strongest perceptual distance among baselines but lacks non-differentiable tool support and tool-calling-style planning outputs (Doh et al., 1 Dec 2025).

The ablations isolate the three main technical ingredients. Removing CoT reduces effect accuracy from 80% to 67% and order correlation from 0.56 to 0.49. Removing Number Token Loss reduces effect accuracy to 73%, worsens parameter MAE to 0.32, and degrades reconstruction metrics. Removing multi-stage training yields 76% effect accuracy and 0.55 order correlation. These results support the paper’s claim that CoT improves planning, number-aware loss improves parameter estimation, and multi-stage training improves multimodal alignment (Doh et al., 1 Dec 2025).

A MUSHRA listening test with 8 questions and 16 participants reports mean scores of 81.5 for the hidden reference, 62.8 for LLM2Fx-Tools, 56.5 for Gemini 2.5 Flash, 54.8 for DeepAFX-ST, 39.1 for No Fx, 34.9 for MultiTask, and 16.2 for Regression. Pairwise tests report that LLM2Fx-Tools significantly outperforms No Fx with $x_{\text{ref}}$1, Gemini 2.5 Flash with $x_{\text{ref}}$2, and DeepAFX-ST with $x_{\text{ref}}$3, while Gemini and DeepAFX-ST are not significantly different with $x_{\text{ref}}$4 (Doh et al., 1 Dec 2025).

In blind estimation / style transfer, LLM2Fx-Tools also leads the reported baselines, with AF 7.41, AFx-Rep 0.35, and Fx-Encoder 0.49. The paper attributes Gemini’s weaker transfer to poor parameter estimation. For natural-language generation judged by GPT-5, LLM2Fx-Tools achieves tool-calling success 99.8%, instruction-following quality 3.50, and CoT quality 3.05, compared with 73.7%, 2.89, and 2.30 for Qwen3-4B, and 100%, 3.39, and 3.03 for Gemini 2.5 Flash (Doh et al., 1 Dec 2025).

The paper’s limitations are explicit. First, interpretability depends on pseudo-dry preprocessing rather than access to the true original dry signal. Second, the mapping from sound to effects chain is one-to-many, so multiple distinct chains may be perceptually similar. Third, the evaluation set is limited in scale. Fourth, experiments are on single-instrument sources rather than full multitrack mixes. Fifth, extension to unseen VSTs or plugins is proposed but not validated. These points align with broader tool-use research showing that executable tool calls do not eliminate uncertainty: system performance depends not only on the model’s generated text but also on the reliability of tool outputs and the degree to which final answers genuinely condition on those outputs (2505.16113).

From a broader research perspective, LLM2Fx-Tools can be read as a domain-specific realization of tool-augmented autoregressive reasoning in which the tool set is the post-production signal chain. This suggests a shift in music-technology research from single-effect semantic control toward executable, interpretable, and controllable multi-tool production workflows. The paper’s broader significance lies less in a new DSP algorithm than in reframing music post-production as structured reasoning plus tool execution over audio effects modules (Doh et al., 1 Dec 2025).