Papers

CLAP: Learning Audio Concepts From Natural Language Supervision

The original CLAP. CNN14 audio tower + BERT text tower projected into a 1024-dim shared space, trained with symmetric InfoNCE on ~128K curated (audio, caption) pairs from FSD50K, ClothoV2, AudioCaps, MACS. Establishes that the CLIP recipe transplants cleanly to audio — and that even at modest scale, you get zero-shot classification across 16 downstream datasets.

Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

The scaled-up successor to MS-CLAP. Swaps CNN14 + BERT for HTSAT + RoBERTa, the shared space drops to 512-dim, and the training set grows ~20x via Keyword-to-Caption augmentation (T5 turns AudioSet labels into synthetic captions) plus LAION-Audio-630K. Attention Feature Fusion handles variable-length clips. This is the CLAP that downstream systems (AudioLDM, Stable Audio, SAM Audio) actually use.

MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training

What CLAP gets via captions, MERT gets from the audio alone. A HuBERT-style transformer masks ~50% of frames and reconstructs from two simultaneous teachers: EnCodec RVQ codes (timbre, texture) and the Constant-Q Transform (pitch, harmony). The dual-teacher design is the load-bearing move — either signal alone leaves the resulting representation blind to half of what music actually is.

Can Large Language Models Predict Audio Effects Parameters from Natural Language?

Can a frozen LLM predict EQ and reverb parameters directly from text descriptions, with no task-specific training? Doh et al. show yes — and three flavors of in-context examples (textual, audio, parameter) push it past prior optimization-based approaches. A surprisingly tractable beachhead for natural-language music production.

LLM2Fx-Tools: Tool Calling for Music Post-Production

Where LLM2Fx asks 'predict the parameter vector,' LLM2Fx-Tools asks 'plan the chain.' A multimodal LLM listens to a target sound, picks the FX modules, orders them, and estimates parameters — emitting tool calls with chain-of-thought planning between steps. Trained on a new instruction-following dataset (LP-Fx) and validated in a style-transfer setting.

When AI don't sound like AI: Negotiating aesthetic expectations in technology-mediated musical practice

What does AI music sound like, really? Pelinski, Melbye, and McPherson revisit their Surfing Hyperparameters project to argue that aesthetic expectations — ghosts in machines, sonic hauntology — actively shape AI music tools rather than emerging from them. Performative storytelling as design practice, not just description.

A Design Space for Live Music Agents

Surveys 184 live-music-agent systems across HCI, AI, and computer music to build a unified design space — usage contexts, interactions, technologies, ecosystems. The map a field-fragmented community has been waiting for, and a tool for spotting corners of the space still unexplored.

Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music

NVIDIA's next-generation open audio-LM doubles down on long-form reasoning: 30-minute inputs and Temporal Audio Chain-of-Thought, where intermediate reasoning steps are grounded to specific timestamps in the audio. Curriculum training plus scalable data construction pushes it past similarly-sized open models across 20 benchmarks.

AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing

Reading audio isn't enough — AudioChat trains a single model to generate, edit, and understand multi-source 'audio stories' (multiple speakers, foreground/background SFX) via a new Audio Transfusion Forcing objective. The clever twist: training data is bootstrapped from LLM-tool-calling agents simulating user-system dialogues, sidestepping the lack of natural multi-turn audio supervision.

The Ghost in the Keys: A Disklavier Demo for Human-AI Musical Co-Creativity

Continuous KV-cache prefill kills the 1–2 second latency that usually breaks real-time human-AI duets. The interface is a Yamaha Disklavier — you play, hit the left pedal to hand over, and Aria's continuation comes back through the same physical piano. The musicological eval across six styles checks for vocabulary consistency, prosodic response, and actual phrasal dialogue.

ReaLJam: Real-Time Human-AI Music Jamming with Reinforcement Learning-Tuned Transformers

Instead of eliminating server latency, hide it: a commitment protocol plus a waterfall display let humans jam chords live with a model over the web. The online Transformer is RL-tuned to anticipate the user's melody from past context only — basically process-reward RLHF for music. The neatest finding is the user study's surprisal–control paradox: musicians who felt more in control rated the session lower.

VampNet: Music Generation via Masked Acoustic Token Modeling

Non-autoregressive audio generation: discrete tokens from the Descript Audio Codec, a bidirectional masked transformer, iterative parallel decoding. The neat part is that vamping, inpainting, and prompt continuation are all the same model — just different mask patterns at inference.

Anticipatory Music Transformer

The symbolic-domain counterpart to VampNet: MIDI as a temporal point process, with control tokens interleaved a fixed interval *before* the events they condition. That anticipation trick keeps the sequence local and gives autoregressive sampling valid stopping times, so you get controllable infilling without ever fine-tuning for the task.

A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation

Stack 8 harmonically shifted copies of a Constant-Q Transform as the input, and a 16,782-parameter CNN can transcribe across instruments. The tradeoff is explicit: give up some single-instrument accuracy for one model that ships everywhere — which is exactly what became Spotify's Basic Pitch.

MT3: Multi-Task Multitrack Music Transcription

Can one seq2seq Transformer transcribe everything? MT3 says yes — given the right token vocabulary (instrument, time, pitch, on/off) and temperature-weighted mixture training so small datasets like GuitarSet and URMP don't get drowned out by MAESTRO.

Onsets and Frames: Dual-Objective Piano Transcription

Onsets are acoustically loud and obvious; sustain frames are mush. So train two specialized heads — CNN into bidirectional LSTMs — and gate the frame predictions on the onset predictions at inference. That single gating rule kills the spurious-sustain failure mode that plagued single-model transcribers.

DDSP: Differentiable Digital Signal Processing

Classical synthesis primitives — oscillator banks, time-varying filters, convolutional reverb — rebuilt as differentiable modules so gradients flow from the audio output all the way back through the synthesis chain. Hard-coding the physics of sound buys a lot: WaveNet-quality audio at roughly 300x fewer parameters.

A Review of Differentiable Digital Signal Processing for Music and Speech Synthesis

Four years of DDSP work, surveyed by the design choices that keep showing up: which primitives to keep differentiable, how to parameterize time-varying filters, what to do about phase. The map we needed to see where the field has actually gone since 2020 — and which problems are still open.