All Notes
Interactive write-ups from past sessions, newest first. 8 notes so far.
Audio Representations
Contrastive language-audio alignment (CLAP) and self-supervised music understanding (MERT)
Two complementary approaches to learning what music 'means' to a neural network: CLAP aligns audio with natural-language descriptions through contrastive learning, while MERT learns musical structure from audio alone via masked acoustic modeling with dual teachers.
LLMs for Audio Effects
Two papers from the Sony AI / KAIST group on using language models to drive EQ, compression, and reverb: first by predicting parameters, then by calling tools
Why audio effects are a uniquely tractable target for LLMs, and how the Doh et al. team moves from parameter regression (LLM2Fx) to tool-calling chains with chain-of-thought planning (LLM2Fx-Tools).
Audio Language Models
From LTU and LLark through Audio Flamingo Next to AudioChat: how audio fuses with text, and where the LM starts writing audio back
How audio is concatenated with text in modern audio LMs, what projection layers do, and what gets frozen at each training stage, across LTU, LLark, Music Flamingo, Audio Flamingo Next, and AudioChat's bidirectional read-and-write extension.
HCI of Live Music Agents
Design spaces, aesthetic expectations, and what AI is actually allowed to sound like on stage
Two papers that step back from architecture and ask different questions: how should we map the space of live music agents (Kim et al.), and how do cultural expectations about what AI 'should sound like' actively shape the systems we build (Pelinski, Melbye, McPherson)?
Real-Time Human-AI Music
Eliminating and masking the two-second latency wall
We examine two 2025 papers that each address the same problem, the approximately two-second latency wall that blocks real-time human-AI musical collaboration, through distinct strategies. Ghost in the Keys eliminates the bottleneck at the inference level via continuous KV-cache prefill on a Yamaha Disklavier. ReaLJam masks it through probabilistic foresight, a commitment protocol, and a waterfall display.
Music Generation
From masked acoustic tokens to anticipatory symbolic infilling
An overview of two approaches to music generation: VampNet's masked acoustic token modeling for audio-domain generation, and the Anticipatory Music Transformer's interleaved infilling for symbolic MIDI.
Music Transcription
Dual-objective CNNs to transformers to 16K-parameter models
An overview of three papers that shaped the trajectory of automatic music transcription, covering Onsets and Frames, MT3, and the lightweight NMP (Basic Pitch) model, with interactive visualizations of the main ideas.
DDSP From Scratch
A minimal differentiable synthesizer in PyTorch (trying to understand the paper)
Building a differentiable digital signal processing synthesizer from scratch, with interactive visualizations and audio experiments from our reading group session on Engel et al. (2020).