HCI of Live Music Agents

Two papers, both stepping back

The previous weeks were architecture-heavy: how the model fuses audio with text, how it predicts the next token, how it stays under the latency budget. This week is an HCI week, and the two papers do something the others don’t. They stop asking how to build the thing and start asking what kind of thing is even being built, and for whom.

The first paper, A Design Space for Live Music Agents (Kim, Brade, Wang, Zhou, Kim, Wang, Lee, Flores Garcia, Huang, Donahue, 2026), maps the territory. The authors survey 184 live music agent systems across HCI, AI, and computer music venues, and propose a unified design space that names the dimensions everyone has been navigating implicitly: what the agent listens to, what it can do, how musicians control it, what it puts back into the room. The map is the contribution, and it’s the kind of paper that should have existed two years ago.

The second paper, When AI don’t sound like AI (Pelinski, Melbye, McPherson, 2026, Organised Sound), zooms way in. The authors reconstruct a single project of theirs, Surfing Hyperparameters (a duo performance system), and use it to argue that aesthetic expectations are not consequences of the technology but active inputs to it. The story we tell ourselves about what AI should sound like ends up shaping the system we build, regardless of what the system would have produced on its own.

Read together, the two papers cover a useful arc. Kim et al. give us the geometry of the field. Pelinski et al. remind us that everyone moving through that geometry is being pushed around by cultural expectations they may or may not have noticed.

A few primers before the deep dives

The first three primers below are about how this week’s papers frame live-music AI. The fourth is about a specific philosophical move (hauntology) that the Pelinski paper leans on heavily and that some readers will not have seen before.

The tool, instrument, agent distinction is the most-cited framing in this week’s papers, so it’s worth making concrete. The interactive below switches between the three modes and shows what changes about how control flows between musician and system.

The three modes don’t differ in technology. They differ in who initiates. A tool waits for the musician to commit. An instrument trades signals continuously. An agent has an internal loop of its own that can lead. By the Kim et al. paper’s bar, almost nothing in current AI music actually clears the agent threshold; most “AI music agents” are sophisticated instruments with very strong priors. That isn’t a failure mode, but it’s a useful thing to know about the field’s actual position.

A design space for 184 live music agents

Kim et al.’s contribution is conceptual. They surveyed 184 systems explicitly designed for live music interaction (real-time exchange with a human performer, not offline composition or passive listening) drawn from CHI, NIME, ICML, arXiv, and adjacent venues, spanning roughly the last two decades with heavy weight on the 2020 to 2026 window. The systems are catalogued along design-space dimensions covering what the agent listens to (audio, symbolic, gestural, contextual signals), what it can do (accompaniment, continuation, real-time transcription, harmonization, sound-design assistance), how the musician communicates with it (real-time MIDI, audio input, gestures, latency budgets), and what it produces (audio synthesis, symbolic streams, control signals to other instruments). The framing extends beyond the obvious technical axes to include usage context, ecosystem, and the social setting around a performance.

The map shows clusters and gaps. Most systems sit on the audio-input / audio-output diagonal, which is partly a tooling artifact: audio is what’s easy to capture and synthesize, and audio-only systems compose cleanly with the rest of the music technology stack. Symbolic and gestural inputs are sparser. Real-time symbolic generation is sparser still; low-latency models tend to be audio-domain, and the symbolic-domain models that exist are mostly offline. Cross-modal agents (listen with audio, respond symbolically; listen with gesture, respond with parameter automation) are largely empty cells.

Two further gaps the authors flag explicitly. First, cultural and genre diversity: training data is dominated by Western classical and jazz traditions, with most systems implicitly inheriting those tonal and rhythmic priors. Second, long-form coherence: the field optimizes hard for short-horizon objectives (the next bar, the next phrase), but live music as practiced runs on much longer arcs of tension and release. Few systems are evaluated for what they do across an entire performance.

A few specific framings the paper makes load-bearing. The “agent” label is reserved for systems that show intentional behavior, meaning systems that adapt strategy mid-performance in response to context, rather than just emitting stochastic output. By that bar, very few current systems are agents; most are sophisticated instruments. The paper doesn’t treat that as a failure. It treats it as a finding about where the field actually is, which is useful to know before claiming to be building something else.

The unaddressed-but-implicit question, which Pelinski et al. take up next, is how a designer ends up at one cell rather than another. The map shows where systems are. It doesn’t say much about why they got there.

Surfing Hyperparameters: when the story shapes the sound

Pelinski, Melbye, and McPherson’s paper is structurally different. Where Kim et al. survey, Pelinski et al. autoethnograph. They take a single project, Surfing Hyperparameters (a performance system Pelinski and Melbye built around a modified double bass called the FAAB, the feedback-actuated augmented bass), and reconstruct, over several years, how cultural narratives about what AI music should sound like repeatedly steered design decisions in particular directions.

The setup is technically grounded. The FAAB has eight piezo contact microphones distributed across the body of the instrument, each capturing a subtly different vibrational signature. Those eight signals feed a Transformer-based autoencoder running on a Bela microprocessor that compresses the input to four latent dimensions. Those four dimensions then drive a modular synthesis chain (Eurorack hardware via the pybela bridge) using a Multi-Scale Spectral Difference (MSSD) sonification scheme that translates latent-space motion into sound. Eight model variations were trained with different hyperparameters, and the team deliberately kept suboptimal versions in the rotation as distinct “sound worlds” rather than collapsing to a single best autoencoder. Performance involved navigating between these models on stage, treating each as a different inhabitable region of the design space.

So far the paper reads like a NIME paper. The contribution comes from what they noticed while doing this. From the paper:

The stories practitioners tell while building and making music with technology are performative, constructing rather than merely describing aesthetic realities.

The argument leans on Lucy Suchman, Bruno Latour, Donna Haraway, and Sharon Traweek. Narratives don’t passively report what happened; they actively produce it. During development, one author observed that the system exhibited “aesthetically indefensible” switching behavior between models, a technical problem by their own framing, and yet they found it “engaging to listen to.” Faced with that contradiction, the post-hoc story they ended up telling justified the sonic outcome through the AI framing, even where the technical reality didn’t quite support it. The MSSD output in the final system, the paper notes, “was not dissimilar to what they imagined the models to sound like when the project was initiated.” Initial expectations shaped final aesthetic satisfaction.

The cultural background hum is sonic hauntology. The paper traces a chain of associations the broader culture brings to AI music: the ghost-in-the-machine trope, the spectral aesthetics of degraded analogue media, the framing of AI as archival haunting (datasets as plundered recordings, resituated in new contexts). These associations, the authors argue, are “conceptual rather than based on a sonic aesthetic,” meaning they’d be applied regardless of what the system actually sounded like. AI sound becomes uncanny because we’ve already decided it should be.

The practical moves the authors recommend are about transparency, not aesthetics. Document expectations, not just developments. Acknowledge mismatches between intended and actual behavior rather than papering over them with conceptual ambiguity (they invoke the Clever Hans effect, where a system appears to do something it doesn’t). Recognize that both technical constraints and cultural narratives shape outcomes; neither operates alone. And practice transparency about the gap between the story and the artifact, so the project as documented matches the project as built.

Mapping Pelinski onto Kim

The two papers don’t reference each other, but they sit next to each other naturally. The Kim et al. design space tells us where in the territory we are. The Pelinski paper tells us why we might be there.

A sample reading: Surfing Hyperparameters sits in a sparsely-populated cell of the Kim map (instrument-domain input from the augmented double bass, continuous low-dimensional latent control over four dims, audio-domain output via modular synthesis, an instrument-rather-than-tool framing, and a live-performance context). By the Kim taxonomy, that cell is fertile and largely empty. By the Pelinski critique, the reason a project ends up in that cell is partly the technology and partly the cultural narrative the team brought into the room. A team that comes in expecting to build something hauntological will tend to inhabit cells where hauntological readings are easy.

That tension generalizes. Wherever the Kim map is empty, the empty region might be empty because the technology can’t reach it yet, or because the cultural narratives that motivate the field don’t currently make that region desirable. Disentangling those two reasons is hard, and the Kim paper doesn’t pretend to. The Pelinski paper offers some methodological grip on the problem: examine the narratives, document the expectations, and separate the story from the artifact. Doing that paper-by-paper would make the design space more honest over time.

Open questions

A few discussion prompts we ended up with prepping this session:

• We’re unsure whether the tool / instrument / agent spectrum is useful for design or only descriptive after the fact. Could a team commit to building an “agent” rather than an “instrument” up front, and have that decision actually shape their training objective?
• Both papers gesture at long-form coherence as under-explored. We don’t have a good answer for what even counts as evaluation for a system that’s supposed to play across thirty minutes of a live set; most metrics in current use are window-of-bars metrics.
• Pelinski et al. argue that aesthetic expectations are inputs to the design process rather than consequences of it. Whether that’s resistible is unclear. A music AI system with no prior aesthetic narrative might still be a stylistic choice in disguise (free-jazz minimalism, Cagean indeterminacy, and so on).
• The Kim design space catalogues 184 systems but doesn’t grade them against each other. That refusal is load-bearing for the framing, but it makes the paper hard to use prescriptively. We’re not sure what concrete advice the paper would give a new team designing a system from scratch.
• We didn’t have time in the session to map this reading group’s other papers (LTU, LLark, AudioChat, Aria-Duet, ReaLJam, the LLM-FX papers from Week 7) onto the Kim design space, but that’s a useful whiteboard exercise.

Papers & resources

Demos & systems referenced