LLMs for Audio Effects

Why audio effects are a great target for LLMs

Music production has a useful set of properties for an LLM to act on. The parameter spaces for individual effects are small: an EQ has a handful of bands, each with frequency, gain, and Q; a compressor has six or seven well-named knobs; a reverb has tail length, dampening, pre-delay, and mix. Every parameter has a name that engineers actually use (“low cut”, “attack”, “ratio”), so language already maps reasonably onto the action space. There is also a very large corpus of producers and engineers describing what effects do in natural language (tutorials, forum posts, mix critiques, gear reviews), which means the priors a frozen LLM brings into this domain are genuinely informed.

The two papers this week are both from the Sony AI, KAIST, and UPenn group around Seungheon Doh, Junghyun Koo, and Marco Martínez-Ramírez. They’re consecutive entries in a research thread, and they’re best read that way. The first paper asks whether a frozen LLM can predict effect parameters directly from text descriptions. The second asks what it looks like to chain those predictions into a production workflow. The first paper is direct prediction. The second paper is tool calling, which turns out to be the more flexible framing.

The papers, in order:

• LLM2Fx (Doh et al., 2025). A frozen LLM predicts EQ and reverb parameters from a natural-language description. Three flavors of in-context examples make it competitive with prior optimization-based approaches.
• LLM2Fx-Tools (Doh et al., 2025). The same group recasts the problem as multimodal tool calling: the LLM listens to audio, plans a chain of effects, and emits structured tool calls with chain-of-thought reasoning between them. Trained on a new instruction-following dataset (LP-Fx) and validated in a style-transfer setting.

This week’s papers also tie back to AudioChat from Week 5. AudioChat used LLM-tool-calling agents to bootstrap training data for an audio LM. LLM2Fx-Tools uses tool calling at inference time, where the tools manipulate effects directly. Same primitive, different deployment.

A few primers

Before the deep dives, two pedagogical detours. The first one shows what an EQ actually does, for readers who’ve never opened a DAW. The second one makes the regression-vs-free-gen-vs-tool-calling distinction concrete.

This is what producers mean when they describe sounds. “More air” pushes the high-shelf knob up. “Warmer” boosts low-shelf and rolls off the top. “Punchy” boosts low-shelf, sometimes with a small upper-mid lift. The labels carry shared meaning, and there’s no perceptual mystery in the action space. That’s part of what makes the LLM’s text-trained priors usable here.

Each of the three formats is a different commitment device. Regression is rigid but compact. Free generation is flexible but brittle. Tool calling is the middle ground that scales, since schemas guarantee well-formed output, the LLM can chain calls, and new effects can be added without retraining. LLM2Fx is closer to free-gen with strict templates. LLM2Fx-Tools is the tool-calling pivot.

LLM2Fx: prompt → parameters, frozen LLM

LLM2Fx (Doh et al., 2025) starts simple. The system has a frozen LLM, a set of audio effects (the paper focuses on equalization and reverberation), and a structured prompt template. Given a natural-language description like “make this guitar warm and intimate” or “thicker low end with airy top”, the LLM generates parameter values for the registered effects, in a parseable format, with no fine-tuning. The LLM never touches audio in the simplest configuration; it works purely text-to-text.

The result that makes the paper non-trivial is that this approach beats prior optimization-based approaches, which iteratively searched parameter space using audio-domain losses against a target. Those baselines were strong (they had access to audio and could optimize directly) but they were also slow because they were iterative, and they required differentiable effect implementations or careful black-box optimizers. LLM2Fx replaces the search loop with a single LLM forward pass and gets competitive results.

The forward pass works because of the in-context examples, which come in three flavors:

• Textual: pairs of (description, parameters) from a curated knowledge base. The LLM sees how producers describe parameter settings and learns to extrapolate.
• Audio: pairs of (audio features, parameters) where the audio features are perceptual descriptors (brightness, warmth, density). This grounds the description in measurable acoustic properties.
• Parametric: bare parameter examples that anchor the numerical scale (e.g., “moderate compression” usually meaning a ratio in the 3:1 to 5:1 range, not 50:1).

The paper ablates these and finds that combining them is what closes the gap to optimization-based methods. Each flavor injects a different prior. Textual examples teach the LLM to map descriptors to parameters. Audio examples link descriptors to acoustic features. Parametric examples calibrate the numerical scale.

The architectural footprint is small. There’s no model training, no audio encoder, no learned projection. The whole system is a prompt template plus a small bank of in-context examples, and it scales by adding examples rather than by training. The deployment story is deliberate: the same setup can absorb new effects (chorus, distortion, delay) by adding an example bank for each, with no retraining required.

The limitations are real and the paper is honest about them. Effect coverage is narrow (EQ, reverb). Continuous parameters are quantized to a vocabulary the LLM can emit. There’s no audio feedback loop, since the LLM commits to parameters once, doesn’t listen to the result, and doesn’t adjust. The output is a single forward pass; iterative refinement has to be wrapped in a loop separately. Most of these limitations are what LLM2Fx-Tools sets out to address.

LLM2Fx-Tools: tool calling for the post-production workflow

LLM2Fx-Tools (Doh et al., Dec 2025, ICLR 2026) recasts the same problem as multimodal tool calling. Instead of emitting parameter strings, the LLM emits tool calls against a registered set of audio-effect modules. Each tool call is a structured JSON payload: pick an effect type, pick its position in the chain, set its parameters. The LLM plans the entire chain via chain-of-thought reasoning between tool calls, and the chain is executable end-to-end.

A few specific deltas from LLM2Fx:

• The LLM consumes audio. It’s a multimodal LLM (audio embedding plus text), so it can listen to the input clip, and in the style-transfer setup also to a reference clip, before planning. Audio goes from “absent” in LLM2Fx to a primary input in LLM2Fx-Tools.
• The output is an executable chain, not a parameter vector. The LLM emits tool calls in order, with the order itself a learned decision (compressor before EQ, reverb last, and so on). The chain runs against a real effect implementation and produces audio.
• Chain-of-thought between calls. The model reasons about each tool call before emitting it (“the target is duller and less compressed than the reference, so add a high-shelf at 4 kHz and a 3:1 compressor with medium attack”). The CoT isn’t only for explainability; it’s part of the trained sequence.
• A new dataset. The paper introduces LP-Fx, an instruction-following dataset built around real music sources (the paper references MedleyDB and MOISESdb), with structured CoT annotations and tool-call traces. Effects are sampled from realistic ranges, so the model sees plausible chains during training rather than uniform-random parameter draws.

The headline experiment is style transfer. Given a reference audio clip (sounds the way it should) and a target audio clip (the unprocessed source), the system has to predict an FX chain that, when applied to the target, makes it match the reference. This task is reasonable because (a) it’s well-defined, with a specific gap to close, and (b) it sidesteps the underspecification of pure language prompts. “Make it warm” can mean many things; “make it sound like that” is much tighter. The paper shows the system can infer non-trivial chains from these reference and target pairs, including the order in which effects should be applied.

The viz below makes the framing concrete. Pick a reference/target relationship (brighter, punchier, warmer) and watch the chain land step by step. The rose-shaded region between the lines is the gap to close, and each tool call eats into it.

Evaluation includes an LLM-as-judge component, where a separate LLM scores the appropriateness of the generated CoT and tool calls against held-out music-production queries. This is an Arena-style eval that sidesteps the absence of a single correct answer in production tasks. There is no ground-truth EQ curve for “make it punchier”, but a competent judge can rank candidates. The paper reports that the system produces appropriate CoT reasoning and tool-call sequences across a range of production queries.

The architectural pivot from regression-style parameter prediction to autoregressive tool calling is what makes the second paper feel like a different system, rather than a bigger version of the first. Tool calling brings four free things. The order is part of the prediction. The model can stop early if a chain is short. Chain-of-thought reasoning composes naturally between calls. And the same primitive scales to dozens of effects without retraining. That’s a lot of architectural mileage from one design choice.

Where the two papers fit together

LLM2Fx and LLM2Fx-Tools land in the same overall area but at different points on a complexity ladder. LLM2Fx is the proof-of-concept: a frozen LLM, in-context examples only, narrow effect coverage, beats optimization baselines on parameter prediction. LLM2Fx-Tools is the productization, where the LLM listens to audio, the action is tool calling, the output is an executable chain, and the task is style transfer.

The arc the two papers describe is general beyond audio FX. Tool calling is likely to win in any domain where the action space is structured, the labels carry semantic content the LLM has prior over, and the user benefit comes from chaining several actions rather than picking one. Audio production is one of the cleanest such domains in music-AI right now, and that’s part of why the second paper exists six months after the first rather than the other way around.

Open questions

A few things we ended up debating prepping the session:

• Both papers stop short of a listening loop, where the LLM would hear the output of its own tool calls and adjust. We’re unsure whether that’s a missing piece or whether the bias-variance trade-off in such a loop is too unfavorable, since closed-loop systems can chase noise. Producers iterate compulsively, but they iterate against their ears, and we don’t yet know what an LLM’s ears would actually contribute.
• The semantic priors LLMs bring to audio FX come from training on text where producers describe effects. That works because production discourse is unusually well-documented. How this approach degrades in domains where the discourse is sparser (mastering, foley, sound design for film) is an open question.
• Tool calling assumes well-defined tools, but the right granularity isn’t obvious. set_eq(band, freq, gain, q) is one extreme; make_brighter(amount) is the other. The paper picks somewhere in between. We don’t have a clean theory of which abstraction level lands the best LLM control.
• The style-transfer framing sidesteps text-prompt ambiguity but introduces a different problem about where the reference audio comes from in a real session. In practice producers do have references, but they’re not always close to the source material in instrumentation or arrangement. We’d want to see an ablation that pushes reference and target apart and reports how the chain prediction degrades.
• LLM2Fx-Tools shares the tool-calling primitive with AudioChat from Week 5 (which used tool-calling agents to bootstrap training data) and arguably with the agent framing from the HCI papers in Week 6. There’s a coherent cross-cutting story about tool-using audio LMs that none of these papers writes explicitly. That’s where this thread may be heading.

Papers & related work

Demos & code