The Two-Second Problem
The experience of playing piano alongside a partner who responds two seconds late is, for most musicians, not simply frustrating but categorically different from musical collaboration. Every phrase hangs unacknowledged; every stylistic cue arrives too late to inform a response. The musical conversation, in Csikszentmihalyi’s (1990) sense of an activity sustained by immediate feedback loops, ceases to exist. We posit that this is the central obstacle facing real-time AI music collaboration as of early 2025.
The bottleneck is structural. Autoregressive transformer models maintain a key-value (KV) cache that stores all prior context for efficient generation. This cache must be computed from scratch each time the model is invoked, an O(N² · d) operation over context length N and model dimension d. For a pianist who has played 30 to 60 seconds of MIDI, this computation requires 1 to 2 seconds before the first output note can be generated, a delay that falls well outside the sub-150ms window within which acoustic feedback is perceived as continuous with the action that caused it.
This session examines two 2025 papers that each address this problem through distinct strategies.
Flow State
Csikszentmihalyi's concept of optimal engaged activity requires immediate feedback. Musical performance depends on sub-150ms response loops, the time window within which acoustic feedback is perceived as continuous with the action that caused it. A two-second gap does not merely slow the interaction; it terminates the conditions under which flow can be maintained.
Musicking
Christopher Small (1998) argued that music is not a noun but a verb, an embodied, social activity of mutual response. Text-based AI music tools are asynchronous by construction: the user describes a desired output, waits, and receives a result. Both Aria-Duet and ReaLJam reject this model. They restore what Small calls *musicking*, in which the human acts, the AI responds, and the exchange itself constitutes the music.
Continuous Prefill
The salient engineering contribution of Aria-Duet is a *continuous prefill* strategy: rather than computing the full KV cache in one operation at handover time, the system divides the growing context into chunks and fills the cache incrementally while the human is still playing. By the time the pedal is pressed, the cache is already warm, reducing the perceived gap from approximately 1.8 seconds to roughly 150ms of speculative re-evaluation.
Commitment Protocol
ReaLJam addresses latency through interface design rather than inference optimization. The AI generates chords several bars ahead, partitioning them into *committed* predictions (immutable, rendered in solid blue) and *uncommitted* ones (flexible, rendered semi-transparently). The human sees committed chords before they play, restoring the anticipatory co-adaptation that makes ensemble performance possible.
Ghost in the Keys: Aria-Duet
Bradshaw et al. (2025) introduce a system that frames human-AI interaction as a conversation between two pianists sharing one instrument. The Yamaha Disklavier functions simultaneously as input device (capturing the human’s MIDI performance) and output device (replaying the AI’s generation acoustically). The human plays a passage, signals a handover by pressing the left pedal, and the model generates a coherent continuation performed on the same physical piano.
The generative model, Aria, is an autoregressive transformer trained on over 100,000 hours of solo piano MIDI from the Aria-MIDI dataset. Two post-training modifications appear salient. First, the model is fine-tuned with explicit sustain pedal tokens, as performers reported that simulated sustain was perceptually jarring. Second, duplicate and near-duplicate pieces are removed from the training corpus to suppress the tendency to complete famous themes verbatim, a behavior that is both musically uninteresting and potentially problematic from a copyright perspective.
Latency Timeline
We encourage playing through both modes to hear the latency difference. The continuous prefill strategy amortizes the O(N² · d) KV cache computation over the human's playing time, distributing it across K chunks rather than computing it in a single operation at handover. The remaining 150ms overhead is for *speculative re-evaluation*: at handover, any notes still sounding have unknown durations. The model predicts a short future of 100 to 200ms to establish correct context before full generation begins.
Beyond the engineering contributions, the paper evaluates musical quality along four dimensions, informed by musicological analysis of the system’s output across six pieces from distinct stylistic traditions:
Semantic consistency concerns whether the AI maintains the musical vocabulary of the piece. In a passage grounded in the Freygish mode, for instance, the AI should not default to Western diatonic patterns; in ragtime, the syncopated 2/4 feel should persist across the AI’s continuation.
Prosody concerns whether the AI follows the micro-improvisations the human introduces. Liszt’s Ave Maria was performed with several expressive variations (e.g., suspensions, rhythmic displacements). The AI’s continuation captured these as part of the contextual vocabulary rather than reverting to the score’s original phrasing.
Musical dialogue concerns whether the AI develops coherent musical responses that reference the human’s phrases and motifs. In the most successful exchanges (the Ave Maria session), the AI maintained three stylistically distinct voices throughout an improvised variation.
Phrasal discourse concerns whether the AI’s phrases exhibit logical shape. Well-formed musical phrases have distinct beginnings (anacrusis), middles, and ends (cadence). The paper reports coherent phrase-level structure across all six test pieces, though we note that the evaluation is conducted by the authors themselves rather than through a blind protocol.
ReaLJam: Real-Time Chord Accompaniment
Scarlatos et al. (CHI 2025) approach the latency problem from the opposite direction. Rather than eliminating the bottleneck at the inference level, they make it perceptually invisible through interface design. The human plays melody over a MIDI keyboard in a web browser; the AI plays accompanying chords. The system addresses three interlocking challenges.
The first is anticipation. The AI generates chords several beats into the future by predicting how the melody will continue. This requires two model variants. The offline model has access to both past and future melody and is used for planning and warm-starts. The online model, which is the variant used at inference time, sees only past melody, but is fine-tuned via reinforcement learning to produce output that appears as though it had access to future context.
The second is communication. The agent’s planned chords are displayed to the human through a waterfall display, a chord grid in which upcoming notes cascade toward a piano keyboard, an interface borrowed from Synthesia and Guitar Hero. Committed chords (immutable after their deadline) appear in solid blue; uncommitted chords that may still adapt appear semi-transparently.
The third is synchronization. A stateless client-server loop ensures chords play at the correct time despite network round-trip latency. The client caches upcoming committed chords while awaiting server responses. As long as the lookahead window in seconds exceeds the round-trip time (at 150 BPM with a one-beat lookahead, the safe margin is approximately 0.32 seconds), no audible delay is perceived.
Anticipation & Lookahead
Toggle lookahead on and off to observe the difference in what the human can anticipate. With lookahead enabled, committed chords (solid blue) are visible two bars ahead of the playhead, and the human can adapt their melody accordingly. Uncommitted chords (purple) may still change based on new input. Without lookahead, the future is opaque.
The RL fine-tuning uses a discriminator D trained to distinguish online from offline output. The reward takes the form:
Two RL variants are evaluated. RL-S assigns rewards at the end of a full chord sequence (a sparse reward signal, analogous to outcome-based RLHF). RL-M assigns rewards at every timestep (a dense reward signal, analogous to process-based RLHF). RL-M shows empirically better local harmonic coherence, which supports the intuition that accompaniment in an interactive setting requires good beat-by-beat decisions rather than globally sensible sequences alone.
How They Compare
We observe that these two papers represent distinct design philosophies applied to the same problem:
| Dimension | Ghost in the Keys (Aria-Duet) | ReaLJam |
|---|---|---|
| Task | Full compositional continuation | Chord accompaniment |
| Hardware | Yamaha Disklavier | Web browser + MIDI |
| Model | Autoregressive, note-level | Transformer + RL, chord-level |
| Latency strategy | Eliminate via KV prefill | Mask via lookahead + commitment |
| User control | Binary (pedal = handover) | Fine-grained UI parameters |
| Training data | 100k hr piano MIDI (Aria-MIDI) | ~30k pop songs (Hooktheory) |
| Evaluation | Musicological analysis (4 dimensions) | User study (experienced musicians) |
The user study in ReaLJam produced one particularly counterintuitive finding: musicians who felt more in control of the AI’s output rated their sessions lower, while those who accepted surprising AI choices rated sessions higher. The authors refer to this as the surprisal-control paradox, and we posit that it raises a genuine question about design goals in creative AI systems. If the aim of an interactive music tool is to maximize user satisfaction, the interface design goal of maximizing user control may in fact be counterproductive. Being surprised, it appears, may be the point.
Discussion Questions
Aria-Duet evaluates output through musicological analysis conducted by the authors themselves, without a blind user study. ReaLJam conducts a user study with experienced musicians but does not include musicological evaluation. Which methodology tells us more about whether a system “works” in the context of human-AI co-creation? We note that the two approaches may be measuring fundamentally different things (i.e., musical quality versus musician satisfaction).
The KV prefill strategy assumes that chunks can be added to the cache incrementally without loss of coherence. Under what conditions does this assumption break down? In particular, what happens if the human pivots style mid-phrase, such that the early cache content is contextually mismatched with the handover context?
ReaLJam’s user study suggests that more control may produce lower satisfaction. Should future AI music tools intentionally reduce user control in favor of surprise? What are the practical and ethical limits of this design direction, and does the finding generalize beyond the specific cohort of musicians studied?
Both models are trained on Western-centric data (classical piano for Aria-MIDI, pop chord progressions from Hooktheory). What happens when a user plays in a non-Western scale, a jazz standard with altered harmony, or a rhythmically complex style that falls outside the training distribution?
Aria-Duet uses a physical acoustic piano as the output device. We posit that the embodied feedback loop (hearing one’s own notes and the AI’s notes in the same acoustic space) may contribute substantially to the perceived quality of the musical dialogue, beyond what the model itself affords. How might one disentangle the contribution of the model from the contribution of the interface?
Both papers address what we might call the last-mile interaction problem, in which the generative model may be sufficiently capable, but the interface between model and musician is the constraining factor. This is a different regime from the Onsets & Frames, MT3, and NMP papers we examined in the previous session, where the model architecture itself was the primary bottleneck. What does this shift suggest about the trajectory of the field?